\chapter {SNNS for Workstation Clusters}
\label{rpcversion}
\index{Workstation-clusters}
\index{RPC Kernel}
\thispagestyle{empty}

Training several neural networks on a single workstation may require a
large amount of training time, if performed on a single workstation.
Dividing the training task among several workstations on a LAN (local
area network) may significantly decrease the overall training time.
This version of SNNS for workstation clusters allows to train several
(possibly different) networks on different workstations and to control
them with a single graphical user-interface.

The kernel interface has been extended for this purpose by an {\em
rpc} module that is controlled by the graphical user-interface xgui.
The module was named {\em rpc} because the communication among the
workstations is performed via the rpc (remote procedure call)
protocol.

This parallel SNNS version supports the following methods of parallelism:
\begin {itemize}
\item training different networks in parallel, each with its own pattern file,
\item training several copies of the same network with different parameters
or different initialization in parallel, each with its own (copied)
pattern file, or
\item an arbitrary combination of the above two methods.
\end {itemize}
The parallel version currently does NOT support dividing the training
of a single network over several workstations. This might be added in
a future version, if it can be implemented efficiently enough.

The extension consists of two programs:
\begin {itemize}
\item {\tt xguirpc}: SNNS simulator program (xgui, simulator kernel,
  and rpc module linked into one executable program). This program is
  the graphical user-interface, which controls the different simulator
  kernels. It has the same functionality as the xgui (snns) program
  plus different panels to control the rpc-interface. To use multiple
  simulator kernels, one may start as many {\tt kernelrpc} programs as
  desired.
\item {\tt kernelrpc}: simulator kernel linked with the rpc module (the
  server). This program is the stand alone simulator kernel with
  rpc module, which is controlled from the {\tt xguirpc} program. Each
  kernel has is own net, pattern sets and configurations.
\end {itemize}

\clearpage

The files are located in the directory 

\hspace*{1.5cm} {$<$snns-root$>$/rpc/bin/$<$architecture$>$}

where $<$architecture$>$ is one of the currently supported machine
architectures. We have tested it on an Intel Paragon and single CPU Sun
SparcStations, and Linux systems. To use the program efficiently on
multi-CPU systems like the DEC Alpha some adjustments in the libraries
might be necessary.

The easiest way to start a new simulator kernel is from the {\bf new
panel} called by pressing the \fbox{\tt NEW} button in the {\bf SNNS
rpc panel} (see \ref{rpcpanel}).

If the networks are to be trained without human supervision, a similar
functionality may also be obtained by starting a copy of the batch
SNNS simulator {\tt snnsbat} on each workstation.


\section {SNNS RPC Programs}

\subsection*{Xguirpc}

{\tt Xguirpc} has the same options as {\tt xgui}. Additionally there
is one more option:

{\tt -force}: \\ The force option is instructing {\tt xguirpc} to
delete the portmap number of an old {\tt xguirpc}. This option is only
necessary if an {\tt xguirpc} was not terminated correctly on the
machine. Only the last registered {\tt xguirpc} will receive messages
from the simulator kernel. This implies that a user can run only one
{\tt xguirpc} on a computer.

\subsection*{Kernelrpc}

{\tt Kernelrpc} has the following options:

\begin {itemize}

\item {\tt uid} $<$xguirpc.user-id$>$: the user-id of the {\tt xguirpc} user, 

\item {\tt host} $<$xgui\_rpc\_host$>$: the host-name of the {\tt xguirpc},

\item {\tt program} $<$xgui\_program\_number$>$: the program number
  under which the {\tt xguirpc} is registered at the portmapper,

\item {\tt version} $<$xgui\_version\_number$>$: the version number
  under which the {\tt xguirpc} is registered at the portmapper,

\item {\tt type} $<$type$>$: kernel type (normal or masters),

\item {\tt net} $<$net-name$>$: the name of a network file that is to be loaded,

\item {\tt pat} $<$pat-name$>$: the name of a pattern file that is to
  be loaded. This option may be given more than once.

\end {itemize}

If {\tt kernelrpc} is started without any options the kernel will wait
a short period of time. If there is no connection to an {\tt xguirpc}
the {\tt kernelrpc} will be aborted. The default waiting time is set
to 15 minutes. The simplest way to start {\tt kernelrpc} is with the
help of the {\bf New panel} of the rpc module, because all options are
set automatically. It is possible to run up to 10 {\tt kernelrpc}s at
the same time on one computer.

In some networks problems occur, caused by the domain name. If NIS
(SUN Yellow pages) is activated, the function {\tt getdomainname} does
not deliver the correct domain name.  However, this problem can be
solved, if the environment variable DOMAINNAME is set to the correct
domain name.

{\tt Example:} On our Sun workstations {\tt getdomainname} returns
{\tt infra} instead of\\ {\tt informatik.uni-stuttgart.de}.  To solve
this problem, we type: {\tt setenv DOMAINNAME
informatik.uni-stuttgart.de}. The best way to achieve this is to add
the above line to the {\tt .cshrc} file.


\section{Control Panels}

The distribution of SNNS onto the workstation cluster and the
distributed execution of the training and recall is controlled via
several windows. The main control panel, the {\bf rpc panel}, is opened by
clicking the rpc button in the manager panel. 


\subsection {SNNS RPC Panel}
\label{rpcpanel}
\index{RPC Panel}

This panel is the control center of the communication module. In this
panel the simulator kernel with which the graphical user-interface
will interact is selected. Choose a row and click \fbox {\tt SWITCH TO},
to specify a simulator kernel to contact. The displays and the control
panel will then be updated.


\begin{figure}[htbp]
  \begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/rpc_panel.ps}
  \end{minipage}
  \end{center}
\caption{The RPC Panel}
\label{rpcpanelpic}
\end{figure}

With this panel the current simulator kernel, shown in the top row, is
chosen. 

The buttons and fields of this panel (shown in figure~\ref{rpcpanelpic})
have the following functionality:

\begin{enumerate}

\item {\bf CURRENT HOST} gives the name of the host computer on which
  the currently selected {\tt kernelrpc} program is running.

\item The {\bf list window} displays the list of all currently registered
  {\tt kernelrpc} programs.

\item \fbox {\tt SWITCH TO} switches to the selected simulator kernel.

\item \fbox{\tt INFO} displays information about the selected
  simulator kernel. No switch of the simulator kernel takes place.
  The information displyed includes: Network names or comments which
  are saved in the network description.

\item \fbox{\tt CLOSE} closes the selected simulator kernel. Before this
  process is completed, the system switches to 'direct local'.  This
  simulator kernel is special because it is not using the standard
  communication protocol.  It is impossible to delete this kernel
  because it is linked directly to the graphical user-interface.  If
  the graphical user-interface is started, the standard simulation
  kernel is 'direct local'. The only difference between a `direct
  local' rpc kernel and the standard SNNS kernel without rpc interface
  is the existence of the \fbox{\tt RPC} button in the {\bf Manager
  panel} with the added rpc functionality.

\item \fbox{\tt SETUP} opens the {\bf Setup panel}. In this panel the
  current settings of the marked kernel are displayed and can be
  modified. These settings are described in section \ref{setup}.

\item \fbox{\tt NEW} creates the {\bf New panel}. With this panel it is
  possible to start new simulator kernels on different computers or to
  link already existing kernels.  The {\bf New panel} is described in
  section \ref{newpanel}.  

\item \fbox{\tt PUT} and \fbox{\tt GET} change the net and the patterns
  within the selected kernel.  

\item \fbox{\tt SHOW INFO} selects the information, which is displayed
  in the {\tt list window}.  

\item \fbox{\tt DONE} closes the {\bf rpc panel}.

\end{enumerate}


\subsection {RPC Setup Panel}
\label {setup}
\index{RPC Setup Panel}

With the help of this panel the current settings of the marked simulator
kernel are displayed and modified.

\begin{figure}[htbp]
  \begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/rpc_setup.ps}
  \end{minipage}
  \end{center}
\caption{The RPC Setup Panel}
\label{rpcsetup}
\end{figure}


\begin{itemize}

\item \fbox{\tt UDP} and \fbox{\tt TCP} switch between the
  connection types UDP and TCP. If the connection type is changed, the
  current connection is terminated and a new connection is
  established. {\tt UDP} and {\tt TCP} are the two possible protocols
  that are build directly upon the internet protocol (IP). {\tt UDP}
  (unreliable datagram protocol) is a little bit faster than {\tt
    TCP}. Since {\tt TCP} is more reliable we recomend it for usage
  with {\tt xguirpc}.

\item \fbox{\tt NORMAL} and \fbox{\tt COOP-MASTER} set the mode of
  the kernel. The function of the coop-master mode has not been
  implemented in this release of the SNNS.

\item {\tt Standard timeout} The waiting interval of the graphical
  user-interface, which is used to determine when an rpc error should
  be displayed.  

\item{\tt Long duration timeout} The long duration timeout field is
  responsible for the waiting time in the learning process.  It
  specifies the time {\tt xguirpc} will wait for a {\tt kernlrpc}
  program to finish a training cycle. If there is no response within
  that period of time {\tt xguirpc} will assume an errror in the
  program execution. This value should be rather large for big nets or
  pattern sets, otherwise an error will be displayed even if there is
  actually none.

\item{\tt Timeout for autoexit} is responsible to specify
  how long a simulator kernel will wait without receiving any messages
  before it automatically stops. This value is entered in minutes. 

\item{\tt switch to local (ALL)} Specifies the threshold for
  asynchronous execution mode. If there are fewer training cycles
  specified in the control panel, {\tt xguirpc} will wait for the
  remote host to finish learning (synchronous mode). If there are more
  training cycles specified, {\tt xguirpc} will start training on the
  remote host and then immediately switch back to `direct local' to
  allow the user further input as long as {\tt kernlrpc} is executing
  training on the remote host ( asynchronous mode). This value is
  relevant only for the training of all patterns.

\item{\tt switch to local (SINGLE)} Same as {\tt switch to local
    (ALL)} but with relevance only to learning a single pattern.
  
\item \fbox{\tt APPLY} All entered values are set only after this button is
  pressed.  

\item \fbox{\tt DEFAULT} will use the default values.

\item \fbox{\tt DONE} closes the window.
\end{itemize}


\subsection {New Panel}
\label{newpanel}
\index{New Panel}

The {\bf New panel} is used to start new simulator kernels on different
computers.  A computer host name is required in the first line (it is also
possible to select one from the list by clicking on it with the left
mouse button). The second line specifies the path to the desired rpc
kernel. Remember to seltect an rpc kernel of the correct computer
architecture, if you are working with a heterogeneous workstation
cluster.  Different calling parameters can be added in the
third line. 

\begin{figure}[htbp]
  \begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/rpc_new.ps}
  \end{minipage}
  \end{center}
\caption{The RPC New Panel}
\label{new-panel}
\end{figure}

The \fbox{\tt NEW} button starts a new rpc kernel.  To start the
program, {\tt xguirpc} uses the `rsh'-instruction to execute a command
on a remote computer system. The newly started simulator kernel
receives the computer name, portmap number and version of the computer
on which the graphical user-interface is running.  After a successful
start of the simulator kernel, the kernel registers with the graphical
user-interface. The kernel is then included in the `host-list' of the
{\bf rpc panel}. 

If the new kernel is not listed in the `host-list' of the {\bf rpc
panel} within a couple of seconds, it might be due to the fact that
the given path is invalid on the specified machine (e.g. wrong mount
prefix). In this case try to determine the correct path and try again.

With the \fbox{\tt RECONNECT} and the \fbox{\tt BROADCAST} buttons, different
already existing simulator kernels can be connected with the graphical
user-interface. The difference between those two buttons is that
\fbox{\tt RECONNECT} selects a specific computer and \fbox{\tt BROADCAST}
sends a signal to all computers in the LAN. It is not guaranteed that
all computers will actually receive the signal.


\subsection {Transfer Panels}
\label{transfer}
\index{Transfer Panels}    

To transfer patterns or networks to or from a local simulator kernel,
the transfer panels can be used.

\begin{figure}[hbtp]
\begin{center}
  \begin{minipage}{0.45\textwidth}
    \screendump{0.6}{xgui_figs/rpc_put.ps}
  \end{minipage}
  \hfill
  \begin{minipage}{0.45\textwidth}
    \screendump{0.7}{xgui_figs/rpc_get.ps}
  \end{minipage}
\end{center}
\caption{The RPC Transfer Panels. Put on the left, Get on the right.}
\end{figure}

If the {\bf Get} panel is chosen, it is necessary to have
`direct-local' as a current host and to have a remote simulator kernel
marked with the mouse. The simulator kernel, which the nets and/or the
patterns will be transfered to, can be chosen with the two
arrows. Note that help files will be created in the directory /tmp as
the transfer takes place. Those files will be deleted after the
transfer.  It is necessary to make sure that enough disk space in /tmp
is available.


\subsection {Show Info Panel}

With this panel it is possible to select the information, displayed in
the {\bf SNNS RPC} panel. Items can be added or deleted with
the two arrows. The order of the selected items can be chosen with
\fbox{\tt UP IN LIST} or \fbox{\tt DOWN IN LIST}.

\begin{figure}[hbtp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{xgui_figs/rpc_show.ps}
  \end{minipage}
\end{center}
\caption{The RPC Info Panel}
\end{figure}

The possible fields are:

\begin {itemize}
\item {\tt HOST-NAME}: the computer name on which the simulator kernel
  is running,

\item {\tt KERNEL-NAME}: the version number under which the
  simulator kernel is registered at the portmapper (this information
  can also be displayed with 'rpcinfo -p'),

\item {\tt STATUS}: the last known state of the simulator kernel,

\item {\tt DOMAINNAME}: the domain name of the network,

\item {\tt HOSTADDRESS}:the host address of the simulator kernel,

\item {\tt K-TYPE}: the type of the simulator kernel ( "coop-master"
  and "normal" are possible, but coop-master is not implemented in the
  current version),

\item {\tt KERNEL\_ID}: id of the simulator kernel in xgui,

\item {\tt CYCLES}: number of cycles the kernel used to train,

\item {\tt EPOCH}: number of epochs the kernel used to train, 

\item {\tt SSE}: the sse error value,

\item {\tt MSE}: the mse error value,

\item {\tt SSE/o-units}: the sse error value divided bv the number of
  output units,

\item {\tt NO.~PATTERN}: the number of patterns, 

\item {\tt NO.~UNITS}: the number of units, 

\item {\tt NO.~INPUT}: the number of input units,

\item {\tt NO.~HIDDEN}: the number of hidden units,

\item {\tt NO.~OUTPUT}: the number of output units,

\item {\tt NET NAME}: the network name,

\item {\tt PATTERN SET NAME}: the name of the pattern sets,

\item {\tt LEARN}: the selected learning parameters,

\item {\tt UPDATE}: the selected update parameters,

\item {\tt INIT}: the selected initialization parameters.

\end {itemize}

Note that all information is usually not displayed immediately, but
with a small delay.


\section {RPC Errors}
\label {rpc-error}
\index{RPC Error}

If an rpc function is called, there may be additional errors besides
regular SNNS errors, such as the call of the {\tt DeletePattern} function
without a defined pattern in the network. Because {\tt rpcgen} is not
generating error code within the rpc routines the result of these
functions has to be checked for errors. Since these routines return a
NULL value in case of an error, the return value can be checked for
NULL and the system can then call an error routine, which determines
the exact cause of the error. The rpc library provides such an
error routine. Because the rpc mechanism can not determine if there is
a net-adapter malfunction or if one of the cables is defect, the
rpc mechanism only determines that there is no response from anyone
within a given time interval. The error cause, delivered by the
error function is not that an adapter card is defect, but that the
time  interval was exceeded (time-out error). This is the most common
type of error and the communication module tries to detect the error
with the help of the graphical user-interface.

To accomplish such a task, the communication module remembers if the
last inquiry was performed synchronously or asynchronously. If the
inquiry was asynchronous the probable cause is that the server is not
ready for a new task, because it has to finish the current one.
Therefore the simulator kernel reports to the graphical
user-interface as soon as it has finished its task. This enables the
communication module to update all status informations about the
simulator kernel. The user will be informed about an error, and if
available what kind of error, over a `pop up' window. 

Due to the separation of the graphical user-interface and the
simulator kernel, the error tolerance has increased. If there is for
example an error in the graphical user-interface, only the interface
will be terminated and after a new start of the graphical
user-interface and its linking to the simulator kernel, the user is
able to proceed exactly where the error occured.

If an error occurs in the kernel the situation is less
complicated. In that case simply a new net has to be started, since
the communication module intercepts all maskable signals and saves the
current net before it ends its operation correctly. 

This kind of independence can be used to train multiple nets in
parallel.  All that has to be done is to load multiple nets with
multiple simulator kernels and to start the training process
asynchronously with the desired parameters. Now the graphical
user-interface can be terminated, while the simulator kernel trains
the nets. It is possible to interrupt the training and to evaluate the
current state of the training process with the graphical
user-interface. After that, the training can be resumed. In case the
simulator kernel has already finished training the net it will wait a
given period of time before it saves the net and ends its operations.
In contrast to the batch version of SNNS, {\tt snnsbat}, it is
possible to select all actions of the kernel from a menu and to
evaluate the current training, e.g. the current epochs and error rate.


     





