\chapter{Using the Graphical User Interface}
\label{XGUIManual}
\thispagestyle{empty}

\index{XGUI} This chapter describes how to use XGUI, the X-Window
based graphical user interface to SNNS\@. XGUI is the usual way to
interact with SNNS on Unix workstations. It tells how to call SNNS and
explains the multiple windows and their buttons and menus. Together
with the chapters~\ref{PatHandle} and~\ref{XGUIEditor} it is probably
the most important chapter in this manual.

\section{Introduction to SNNS}
SNNS is a very comprehensive package for the simulation of neural
networks. It may look a little daunting for first time users. This
section is intended as a quick starter for using SNNS. Refer to the
other chapters of this manual for more detailed information.

Before using SNNS your environment should be changed to include the
relevant directories. This is done by :
\begin{enumerate}
\item copy the file SNNSv4.1/default.cfg to your favorite directory

\item copy the file SNNSv4.1/help.hdoc to your favorite directory

\item Set the environment variable XGUILOADPATH to this directory with
  the command {\tt setenv XGUILOADPATH your\_directory\_path}.  You
  could add this line to your {\tt .login} file, so that the help and
  configuration files are available whenever SNNS is started.
\end{enumerate}

\subsection{Startup}
SNNS comes in two guises: It can be used via an X-windows user
interface, or in 'batch mode', that is without user interaction. To
run it with the X-GUI, type {\tt snns}. You obviously need an
X-terminal. The default setting for SNNS is to use colour screens, if
you use a monochrome X-terminal start it up using {\tt snns -mono}.
You will loose no functionality - some things are actually clearer in
black and white.

After starting the package a banner will appear which will vanish
after you click the left mouse button in the panel. You are then left
with the SNNS manager panel.

\begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.7}{xgui_figs/snnsmngr2.eps}
  \end{minipage}
  \caption{The SNNS manager panel}
  \label{fig:mngr}
 \end{center}
\end{figure}

The SNNS manager allows you to access all functions offered by the
package. It is a professional tool and you may find it a little
intimidating. You will not need to use the majority of the options.
You should read this introduction while running the simulator - the
whole thing is quite intuitive and you will find your way around it
very quickly.
 
\subsection{Reading and Writing Files}
SNNS supports five types of files, the most important ones are:
 \begin{description}
\item[\fbox{NET}] Network definition files containing information on
  network topology and learning rules. The files end in the extension
  '.net'. 
\item[\fbox{PAT}] Pattern files, containing the training and test
  data. All pattern files end in '.pat'. 
\item[\fbox{RES}] Results files. Network output is interpreted in many
  possible ways, depending on the problem. SNNS allows the user to
  dump the network outputs into a separate file for later analysis.  
\end{description}
The final two file types are not important for a first exploration of SNNS. 
 
The first thing you are likely to use is the \fbox{FILE} option in the
manager panel to read network and pattern definition files. The window
that will appear is given in figure \ref{fig:file}.

\begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.7}{xgui_figs/snnsfile2.eps}
  \end{minipage}
  \caption{The SNNS file browser}
  \label{fig:file}
 \end{center}
\end{figure}

The top text field shows the current directory. The main field shows
all files for each of the file types that SNNS can read/write.
Directories are marked by square brackets. To load an example network
change the directory by entering the example directory path in the top
field (do not press return):

{\tt SNNSv4.1/examples}

Changes will only be apparent after one of the file selectors has been
touched:click on \fbox{PAT} and then \fbox{NET} again. You should now
see a list of all network definition files currently available.
Double-clicking on one of the filenames, say 'letters' will copy the
network name into the file name window. To load the network simply
click on \fbox{LOAD}. You can also enter the filename directly into the
file name window (top left).

\subsection{Creating New Networks}
You will need to create your own networks. SNNS allows the creation of
many different network types. Here is an example of how to create a
conventional (fully connected) feed-forward network.
 
First select the FEEDFORWARD option, hidden under the \fbox{BIGNET}
button in the manager panel. You are then faced with the panel in
figure \ref{fig:bignet}. Only two parts of the panel are required.

The top allows the definition of the network topology, that is how
many units are required in each layer and how they should appear if
the network is displayed on the screen. The lower part allows you to
fully connect the layers and to create the network.

Note that much of what you are defining here is purely cosmetic. The
pattern files contain a given number of inputs and outputs, they have
to match the network topology; how they are arranged in the display is
immaterial.

 \begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.5}{xgui_figs/snnsbignet2.eps}
  \end{minipage}
  \caption{The SNNS BigNet Feedforward network designer panel}
  \label{fig:bignet}
 \end{center}
\end{figure}

First you have to define the input layer by filling the blanks in the
top right hand corner (edit plane) of the panel.  The panel shows the
current settings for each group of units (a plane in SNNS
terminology).  Each group is of a given type (i.e. input, hidden or
output), and each plane contains a number of units arranged in an
x-y-z coordinate system. This is used for drawing networks only!

You can change the entries by entering values into the boxes or by
clicking on \fbox{TYPE} and \fbox{POS} to change the unit type and relative
position. The relative position is not used for the first plane of
units (there is nothing to position it relatively to). The layers
will, for instance, be positioned below the previous layers if the
'Rel. Position' has been changed to 'below' by clicking on the \fbox{POS}
button.

Here is an example of how to create a simple pattern associator
network with a 5x7 matrix of inputs inputs, 10 hidden units and 26
outputs:
\begin{verbatim}
Leave 'Type' as input
set no 'x' direction to 5
set no 'y' direction to 7
and click on \fbox{ENTER} 
\end{verbatim}
If the input is acceptable it will be copied to the column to the
left. The next step is to define the hidden layer, containing 10
units, positioned to the right the inputs.
\begin{verbatim}
Change 'Type' from input to hidden by clicking on \fbox{TYPE} once.
set no 'x' direction to 1
set no 'y' direction to 10
change 'Rel.Position' to 'below' by clicking on \fbox{POS}
and click on \fbox{ENTER} 
\end{verbatim}
You are now ready to define the output plane, here you want 26 output
units to the right of the input. You may want to save space and
arrange the 26 outputs as two columns of 13 units each.
\begin{verbatim}
Change 'Type' from hidden to output by clicking on \fbox{TYPE} again.
set no 'x' direction to 2
set no 'y' direction to 13
and click on \fbox{ENTER} 
\end{verbatim}
After defining the layer topology the connections have to be made.
Simply click on \fbox{FULL CONNECTION} (bottom left of lower panel). Then
select \fbox{CREATE NET} and \fbox{DONE}. You may have to confirm the
destruction of any network already present.

Selection of \fbox{DISPLAY} from the SNNS-manager panel should result in
figure \ref{fig:display}.

\begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.3}{xgui_figs/display.eps}
  \end{minipage}
  \caption{SNNS network display panel}
  \label{fig:display}
 \end{center}
\end{figure}

The lines showing the weights are not normally visible, you have to
switch them on by select \fbox{SETUP}, and then click on the \fbox{ON} button
next to 'links' option.  You will find that SNNS refuses any further
input until you have selected \fbox{DONE}.

After creating the network and loading in the pattern file(s) - which
have to fit the network topology - you can start training the net. The
network you have just created should fit the 'letters' patterns in the
SNNS examples directory.

An alternate way to construct a network is via the graphical network
editor build into SNNS. It is best suited to alter an existing large
network or to create a new small one. For the creation of large
networks use bignet. The network editor is described in
chapter~\ref{XGUIEditor}.

\subsection{Training Networks}
Load the 'letters' pattern file (in the SNNS-{\tt examples} directory)
at this stage. The network is a pattern associator that can be trained
to map an input image (5x7 pixel representation of letters) into
output units where each letter is represented by an output unit.

All training and testing is done via the \fbox{CONTROL} option. I will
discuss the most important features one by one. The panel consists of
two parts. The top part controls the parameters defining the training
process, the bottom three rows are blanks that have to be filled in to
define the learning rates etc. and the range over which weights will
be randomly distributed when the network is initialised. The defaults
for the learning parameters are (0.2 0 0 0 0) while the default weight
setting is between 1 and -1 (1.0 -1.0 0 0 0).

\subsubsection{Initialisation}
Many networks have to be initialised before they can be used. To do
this click on \fbox{INIT} (top line of buttons in control). You can change
the range of random numbers used in the initialisation by entering
appropriate values into the fields to the right of 'INIT' at the lower
end of the control panel.

\begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.5}{xgui_figs/snnsctrl2.eps}
  \end{minipage}
  \caption{SNNS network training and testing control panel}
  \label{fig:remote}
 \end{center}
\end{figure}

\subsubsection{Selecting a learning function} 
The default learning function for feed-forward nets is
Std\_Backpropagation, you may want something a little more
extravagant. Simply click on \fbox{SEL FUNC} (Select function next to the
learning parameters, see figure \ref{fig:remote}) and pick what you
want to use. The routines you may want to consider are
Std\_Backpropagation, BackpropMomentum or Rprop).  Use
BackpropMomentum for the letters example.

Each learning function requires a different parameter set: here are
the important ones, details are given in the manual:
\begin{description}
\item[Std\_Backpropagation] 1: $\alpha$ learning rate (0-1), 2: $
  d_{max} $, the maximum error that is tolerated. use 0 or a small
  value.
\item[BackpropMomentum] 1: $\alpha$ learning rate (0-1), 2: $\mu$
  momentum term (0-0.99), 3: $c$ flat spot elimination (ignore) and 4:
  $d_{max}$ max ignored error.
\item[Rprop] 1: starting values of $\Delta_{ij}$ (0-0.2) 2:
  $\Delta_{max}$ maximum update value (30 works well..) 3: $\alpha$
  the weight decay term as an exponent (5 works for most problems) $x
  = 10^{-\alpha} = 0.00001$.
\end{description}


Once all parameters are set you are ready to do some training.
Training is done for a number of 'CYCLES' or epochs (enter a number,
say 200 - see fig. \ref{fig:remote}). All training patterns are
presented once during each cycle. It is sometimes preferable to select
the patterns randomly for presentation rather than in order: Click on
\fbox{SHUFFLE} to do this.

For the pattern associator example leave the learning rate at 0.2 and
set the momentum term (second field) to 0.5; leave everything else at
0.

Before starting the learning process you may like to open a GRAPH panel
(from \fbox{manager}) to monitor the progress during training.\footnote{ If
  you do this scale the y-range to lie between 0 and 26 by clicking on
  the 'right-arrow' next the 'Scale Y:' a few times. You can also
  resize the window containing the graph.}
 
Click on \fbox{ALL} to start training and \fbox{STOP} to interrupt training at
any time. The graph will start on the left whenever the network is
initialised so that it is easy to compare different learning
parameters. The current errors are also displayed on the screen so
that they could be used in any graph plotting package (like xmgr).

It is impossible to judge the network performance from the training
data alone.  It is therefore sensible to load in a 'test' set once in
a while to ensure that the net is not over-training and generalising
correctly.  There is no test set for the letters example. You can have
up to 5 different data sets active at any one time. The two \fbox{USE}
buttons on the control panel allow you to select which data sets to
use for training and validation. The top button selects the training
set, the bottom one the 'validation set'. If you enter a non-zero
value into the box next to \fbox{VALID} a validation data set will be
tested and the root-mean-square error will be plotted on the graph in
red every N cycles (N is the number you entered in the box).

You can also step through all the patterns in a data set and, without
updating any weight, calculate the output activations. To step through
the patterns click on \fbox{TEST}.

You can go to any pattern in the training data set by either
specifying the pattern number in the field next to 'PATTERN' and
clicking on \fbox{GOTO} or by using the 'tape player controls' positioned
to the right of \fbox{GOTO}. The outputs given by the network when stepping
though the data are the targets, not the calculated outputs (!).


\subsection{Saving Results for Testing}
Network performance measures depend on the problem. If the network has
to perform a classification task it is common to calculate the error
as a percentage of correct classifications. It is possible to tolerate
quite high errors in the output activations.  If the network has to
match a smooth function it may be most sensible to calculate the RMS
error over all output units etc.

The most sensible way to progress is to save the output activations
together with target values for the test data and to write a little
program that does whatever testing is required. The \fbox{RES} files under
\fbox{FILE} are just the ticket: Note that the output patterns are always
saved. The 'include output patterns' actually means 'include target
(!) patterns'.

\subsection{Further Explorations}
It is possible to visualise the weights by plotting them, just like
the output values as boxes of different sizes/colour. Sometimes
examining the weights gives helpful insights into how the networks
work. Select \fbox{WEIGHTS} from the manager panel to see the weight
diagram.

\subsection{SNNS File Formats}
\subsubsection{Pattern files}
To train a network on your own data you first have to massage the data
into a format that SNNS can understand. Fortunately this is quite
easy. SNNS data files have a header component and a data component.
The header defines how many patterns the file contains as well as the
dimensionality of the input and target vectors. The files are saved as
ASCII test. An example is given in figure \ref{fig:ffmt}.

\begin{figure}[htbp]
 \begin{center}
  \begin{minipage}[htbp]{\textwidth}
   \screendump{0.7}{xgui_figs/snnsffmt.eps}
  \end{minipage}
  \caption{Pattern file diagram}
  \label{fig:ffmt}
 \end{center}
\end{figure}

The header has to conform exactly to the SNNS format, so watch out for
extra spaces etc. I found it easiest to copy headers from one of the
example pattern files and to edit the numbers. The data component of
the pattern file is simply a listing of numbers that represent the
activations of the input and output units. For each pattern the number
of values has to match the number of input plus thenumber of output
units of the network as defined in the header. For clarity you may
wish to put comments (lines starting with a hash (\#)) between your
patterns like shown in figure \ref{fig:ffmt}. They are ignored by SNNS
but may be used by some pattern processing tools. The pattern
definitions may have 'CR' characters in them.

Note that while the results saved by SNNS use (almost) the same file
format as used for the pattern files, the label values defined in the
pattern files are not used.

\subsubsection{Network files}
The networks files, just as the pattern and result files are stored as
ASCII files, they are relatively easy to read and you may find it
easier to hand-edit the network definition file than to use the
graphical user interface to perform tasks such as changing the unit
transfer function or to change the network topology.


\section{XGUI Files}

The graphical user interface consists of the following files:
\smallskip

\index{Default.cfg}
\index{Help.hdoc}
\begin{tabular}{ll} 
{\tt snns} or {\tt xgui} 
                     & SNNS simulator program (XGUI and simulator kernel\\
                     & linked together into one executable program)\\
{\tt default.cfg}    & default configuration (see chapter~\ref{FilePanel})\\
{\tt help.hdoc}      & help text used by XGUI\\
\end{tabular}
\bigskip

The file {\tt snns} in the home directory of SNNS is only a symbolic
link to the file 

\hspace{2cm}{\tt xgui/bin/<architecture>/xgui}

where {\tt <architecture>} is one of the currently supported machine
architectures, like {\tt dec\_alpha}, {\tt dec\_mips}, {\tt hp\_ux},
{\tt ibm\_aix}, {\tt sgi\_irix}, {\tt pc\_linux}, {\tt sun\_os4x}, or
{\tt sun\_solaris}.

The file {\tt Readme\_xgui} contains changes performed after printing
of this document. The user is urged to read it, prior to using
XGUI\@. The file  {\tt help.hdoc} is explained in
chapter~\ref{HelpWindows}.

XGUI looks for the files {\tt default.cfg} and {\tt help.hdoc} first
in the current directory. If not found there, it looks in the
directory specified by the environment variable {\tt XGUILOADPATH}. By
the command

\hspace{2cm}{\tt setenv XGUILOADPATH} {\sl Path}

this variable can be set to the path where {\tt default.cfg} and {\tt
help.hdoc} are located.  This is best done by an entry to the files
{\tt .login} or {\tt .cshrc}. Advanced users may change the help file
or the default configuration for their own purposes. This should be
done, however, only on a copy of the files in a private directory.

\index{File Extension}
SNNS uses the following extensions for its files:

\begin{tabbing}
\hspace{2cm} \= {\tt .netxxxxx} \=              \kill
             \> {\tt .net}      \> network files (units and link weights)\\
             \> {\tt .pat}      \> pattern files\\
             \> {\tt .cfg}      \> configuration settings files\\
             \> {\tt .txt}      \> text files (log files)\\
             \> {\tt .res}      \> result files (unit activations)\\
\end{tabbing}

\index{Start Option}
A simulator run is started by the command 

\hspace{1cm}{\tt snns [<netfile>.net] [<pattern>.pat] [<config>.cfg]
[options]} \fbox{\tt Return}

where valid options are

\begin{quote}
-font  {\tt <name>} : font for the simulator\\
-dfont {\tt <name>} : font for the displays\\
-mono : black \& white on color screens\\
-help : help screen to explain the options
\end{quote}

in the home directory of SNNS or by directly calling 

\hspace{2cm}{\tt <SNNS-directory>/xgui/bin/<architecture>/xgui} 

from any directory.  Note that the shell variable XGUILOADPATH must be
set properly before, or SNNS will complain about missing files {\tt
default.cfg} and {\tt help.hdoc}.

The executable {\tt xgui} may also be called with X-Window parameters as
arguments.

Setting the display font can be advisable, if the font selected
by the SNNS automatic font detection looks ugly. The following example
starts the display with the 7x13bold font

\hspace{2cm}{\tt snns -font 7x13bold} \fbox{\tt Return}

The fonts which are available can be detected with the
program xfontsel (not part of this distribution).
The current version of SNNS can not handle fonts wider than 8 pixels. 


\section{Windows of XGUI}
\index{XGUI!Window}
\index{Windows of XGUI}

The graphical user interface has the following windows, which can be
positioned and handled independently ({\em toplevel shells}\/):

\begin{itemize}

\item {\bf Manager} panel with buttons to open other windows, a
  message line, and a line with status information at the bottom .

\item {\bf File} browser for loading and saving networks and pattern
files.

\item {\bf Control} panel for simulator operations.

\item {\bf Info} panel for setting and getting information about unit
  and link attributes. 

\item several {\bf Displays}, to display the network graphically in
two dimensions.

\item {\bf 3D View} panel to control the three dimensional network
visualization component.

\item {\bf Graph} display, to explain the network error during
teaching graphically.

\item {\bf Bignet} panel to facilitate the creation of big regular
networks.

\item {\bf Pruning} panel for control of the pruning algorithm.

\item {\bf Cascade} panel for control of the learning phase of cascade
correlation learning.

\item {\bf Kohonen} panel, an extension to the control panel for
  Kohonen networks. 

\item {\bf Weight Display}, to show the weight matrix as a WV- or
Hinton diagram.

\item {\bf Projection} panel to clarify the influence of two units onto
  a third one.

\item {\bf Analyzer} for supervising recurrent (and other) networks. 

\item {\bf Inversion} display, to control the inversion method network
  analyzing tool.

\item {\bf Print} panel to generate a PostScript picture of one of the
  2D displays.

\item {\bf Help windows} to display the help text.

\end{itemize}

Of these windows only the Manager panel and possibly one or more 2D
displays are open from the start, the other windows are opened with
the corresponding buttons in the manager panel or by giving the
corresponding key code while the mouse pointer is in one of the SNNS
windows.

Additionally, there are several popup windows ({\em transient
  shells}\/) which only become visible when called and block all other
XGUI windows. Among them are various {\bf Setup} panels for
adjustments of the graphical representation. (called with the button
\fbox{\tt SETUP} in the various windows)

There are a number of other popup windows which are invoked by
pressing a button in one of the main windows or choosing a menu.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.75}{xgui_figs/ManagerControlDisplay.ps}
  \end{minipage}
\caption{Manager panel, info panel, control panel and a display.}
\label{ManagerControlDisplay}
\end{center}
\end{figure}


Figure~\ref{ManagerControlDisplay} shows a typical screen setup. The
{\bf Manager} panel contains buttons to call all other windows of the
interface and displays the status of SNNS. It should therefore always
be kept visible.  

The {\bf Info} panel displays the attributes of two units and the data
of the link between them. All attributes may also be changed here. The
data displayed here is important for many editor commands.

In each of the  {\bf Displays} a part of the network is displayed,
while all settings can be changed using  {\bf Setup}. These windows
also allow access to the {\bf network editor} using the keyboard (see
also chapter \ref{XGUIEditor}).

The {\bf Control} panel controls the simulator operations during
learning and recall.

In the  {\bf File} panel a log file can be specified, where all XGUI
output to  {\tt stdout} is copied to. A variety of data about the
network can be displayed here. Also a record is kept on the load and
save of files and on the teaching.

\index{Help!Text}
The complete help text from the file {\tt help.hdoc} is available in
the text section of a {\bf help window}. Information about a word can
be retrieved by marking that word in the text and then clicking
\fbox{\tt LOOK} or \fbox{\tt MORE}. A list of keywords can be obtained
by a click to \fbox{\tt TOPICS}. This window also allows context
sensitive help, when the editor is used with the keyboard. 

{\tt QUIT} is used to leave XGUI\@. XGUI can also be left by pressing
ALT-q in any SNNS window. Pressing ALT-Q will exit SNNS without asking
further questions.

\subsection{Manager Panel}
\label{ManagerPanel}
\index{Manager Panel}
\index{Panel!Manager Panel}


Figure~\ref{FigManagerPanel} shows the manager panel. From the
manager panel all other elements that have a different, independent
window assigned can be called. Because this window is of such central
importance, it is recommended to keep it visible all the time.

\begin{figure}[bthp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.65}{xgui_figs/FigManagerPanel.ps}
  \end{minipage}
\caption{Manager panel}
\label{FigManagerPanel}
\end{center}
\end{figure}

The user can request several displays or help windows, but only one
control panel or text window. The windows called from the manager panel
may also be called via key codes as follows (`Alt-' meaning the
alternate key in conjunction with some other key).


\begin{center}
\begin{tabular}{lrp{1cm}lr}
\fbox{\tt FILE~~~~~~~}&{\tt Alt-f}&&\fbox{\tt CONTROL~~~~}&{\tt Alt-c}\\[0.3ex]
\fbox{\tt INFO~~~~~~~}&{\tt Alt-i}&&\fbox{\tt DISPLAY~~~~}&{\tt Alt-d}\\[0.3ex]
\fbox{\tt 3D DISPLAY~}&{\tt Alt-3}&&\fbox{\tt GRAPH~~~~~~}&{\tt Alt-g}\\[0.3ex]
\fbox{\tt BIGNET~~~~~}&{\tt Alt-b}&&\fbox{\tt PRUNING~~~~}&{\tt }\\[0.3ex]
\fbox{\tt CASCADE~~~~}&{\tt }&     &\fbox{\tt KOHONEN~~~~}&{\tt Alt-k}\\[0.3ex]
\fbox{\tt WEIGHTS~~~~}&{\tt Alt-w}&&\fbox{\tt PROJECTION~}&{\tt Alt-p}\\[0.3ex]
\fbox{\tt ANALYZER~~~}&{\tt Alt-a}&&\fbox{\tt INVERSION~~}&{\tt }\\[0.3ex]
\fbox{\tt PRINT~~~~~~}&{\tt }&     &\fbox{\tt HELP~~~~~~~}&{\tt Alt-h}\\[0.3ex]
\fbox{\tt QUIT~~~~~~~}&{\tt Alt-q}&&&\\
\end{tabular}
\end{center}

SNNS Status Message:
\index{Manager Panel!Status Message}

This line features messages about a current operation or its
termination. It is also the place of the command sequence display of
the editor.  When the command is activated, a message about the
execution of the command is displayed. For a listing of the command
sequences see chapter~\ref{XGUIEditor}.
 
Status line
\index{Manager Panel!Status Line}

This line shows the current position of the mouse in a display, the
number of selected units, and the position of flags, set by the
editor.

X:0 Y:0 gives the current position of the mouse in the display in SNNS
unit coordinates.

The next icon shows a small selected unit. The corresponding value is
the number of currently selected units. This is important, because
there might be selected units not visible in the displays. The
selection of units affects only editor operations (see
chapter~\ref{XGUIEditor} and \ref{Mouse}).

The last icon shows a minature flag. If {\tt safe} appears next to the
icon, the safety flag was set by the user (see
chapter~\ref{XGUIEditor}). In this case XGUI forces the user to
confirm any delete actions.


\subsection{File Browser}
\label{FilePanel}
\index{File Browser Panel}
\index{Panel!File Browser Panel}

The file browser handles all load and save operations of networks,
patterns, configurations, and the contents of the text window.
Configurations include number, location and dimension of the displays as
well as their setup values and the name of the layers.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{xgui_figs/FigFilePanel.ps}
  \end{minipage}
\caption{File Panel.}
\label{FigFilePanel}
\end{center}
\end{figure}

In the top line, the path ({\em without} trailing slash) where the
files are located is entered. This can be done either manually, or by
double--clicking on the list of files and directories in the box on
the left. A double click to [..] deletes the last part of the path,
and a double click to a subdirectory appends that directory to the
path. In the input field below the path field, the name for the
desired file (without extension) is entered. Again, this can be done
either manually, or by double--clicking on the list of files in the
box on the left. Whether a pattern file, network file, or other file is
loaded/saved depends on the settings of the corresponding buttons
below. With the setting of picture~\ref{FigFilePanel} a network file
would be selected. A file name beginning with a slash (/) is taken to
be an absolute path.

Note: The extension {\tt .net} for nets, {\tt .pat} for patterns, {\tt
.cfg} for configurations, and {\tt .txt} for texts is added
automatically and {\em must not} be specified. After the name is
specified the desired operation is selected by clicking either
\fbox{\tt LOAD} or \fbox{\tt SAVE}.
\index{Load!File}
\index{Save!File}
In the case of an error the confirmer appears with an appropriate
message. These errors might be:


\begin{center}
\begin{tabular}{ll}
Load: & The file does not exist or has the wrong type\\
Save: & A file with that name already exists\\
\end{tabular}
\end{center}

Depending upon the error and the response to the confirmer, the action
is aborted or executed anyway.

NOTE: The directories must be executable in order to be processed
properly by the program!

\subsubsection{Loading and Saving Networks}
\label{NetworkSaving}
\index{Load!Network File}
\index{Save!Network File}

If the user wants to load a network which is to replace the net in
main memory, the confirmer appears with the remark that the current
network would be erased upon loading. If the question {\tt 'Load?'} is
answered with \fbox{\tt YES}, the new network is
loaded. The file name of the network loaded last appears in the window
title of the manager panel.

Note 1: Upon saving the net the kernel compacts its internal data
structures if the units are not numbered consecutively. This
happens if units are deleted during the creation of the network. All
earlier listings with unit numbers then become invalid. The user is
therefore advised to save and reload the network after creation, before
continuing the work.

Note 2: The assignment of patterns to input or output units may be
changed after a network save, if an input or output unit is deleted
and is inserted again. This is caused because the activation values in
the pattern file are assigned to units in ascending order of the unit
number. However, this order is no longer the same because
the new input or output units may have been assigned higher unit
numbers than the existing input or output units. So some components of
the patterns may be assigned incorrectly. 


\subsubsection{Loading and Saving Patterns}
\index{Load!Pattern File} \index{Save!Pattern File} Patterns are
combinations of activations of input or output units.  Pattern files,
like nets, are handled by the SNNS kernel. Upon loading the
patterns, it is not checked whether the patterns fit to the network. If
the number of activation values does not fit to the number of input,
resp. output units, a sub-pattern shifting scheme has to be defined
later on in the sub-pattern panel. See chapter~\ref{PatHandle} for
details. The filename of the patterns loaded last is displayed in the
control panel.

Note: The activation values are read and assigned to the
input and output units sequentially in ascending order of the unit
numbers (see above).


\subsubsection{Loading and Saving Configurations}
\index{Load!Configuration File}
\index{Save!Configuration File}

A configuration contains the location and size of all displays
with all setup parameters and the names of the various layers. This
information can be loaded and saved separately, since it is
independent from the networks. Thereby it is possible to define one
configuration for several networks, as well as several configurations
for the same net. When XGUI is started, the file {\tt default.cfg} is
loaded automatically.
\index{Default.cfg}

\subsubsection{Saving a Result file}
\index{Save!Result File}

\begin{figure}[hbtp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.6}{xgui_figs/xgui_result.ps}
  \end{minipage}
\caption{Result File Popup}
\label{resultPopup}
\end{center}
\end{figure}

A result file contains the activations of all output units. These
activations are obtained by performing one pass of forward
propagation. After pressing the \fbox{\tt SAVE} button a popup window
lets the user select which patterns are to be tested and which
patterns are to be saved in addition to the test output.
Picture~\ref{resultPopup} shows that popup window. Since the result
file has no meaning for the loaded network a load operation is not
useful and therefore not supported. 

\subsubsection{Defining the Log File}
\index{Log File}

Messages that document the simulation run can be stored in the {\bf
log file}. The protocol contains file operations, definitions of values
set by clicking the \fbox{\tt SET} button in the info panel, as well
as a teaching protocol (cycles, parameters, errors). In addition, the
user can output data about the network to the log file with the help
of the info panel. If no log file is loaded, output takes place only
on {\tt stdout}. If no file name is specified when clicking \fbox{\tt
LOAD}, a possibly open log file is closed and further output is
restricted to {\tt stdout}.
  

\subsection{Control Panel}
\index{Control Panel}
\index{Panel!Control Panel}
\label{remPanel} 

With this window the simulator is operated.  Figure~\ref{ControlPanel}
shows this window.  Table~\ref{TabControlPanelItem} lists all the input
options with types and value ranges. The meaning of the 5 learning
parameters depends upon the learning function selected with the menu
{\tt select learning function} invoked by the button \fbox{\tt
  OPTIONS} of the control panel.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/ControlPanel.ps}
  \end{minipage}
\caption{Control Panel}
\label{ControlPanel}
\end{center}
\end{figure}
\vspace*{5mm}
\begin{table}[htbp]
\begin{center}
\begin{tabular}{|l|c|c|}
\hline
Name                            & Type  & Value Range\\ \hline
{\tt STEPS} (update-Steps)            & Text  & $0 \le n$\\
{\tt COUNT} (counter for steps)       & Label & $0 \le n$\\
{\tt CYCLES}                          & Text & $0 \le n$\\
{\tt PATTERN} (number of current pattern)                     & Label & $0 \le n$\\
{\tt VALID}                           & Text & $0 \le n$\\
{\tt LEARN} (5 parameters: $\eta$, $\alpha$, $d$, \ldots)     & Text & float\\
{\tt UPDATE} (5 parameters)                                   & Text & float\\
{\tt INIT} (5 parameters)                                   & Text & float\\
\hline
\end{tabular}
\end{center}
\caption{Input fields of the control panel}
\label{TabControlPanelItem}
\end{table}

There are the following text fields, buttons and menu buttons:

\begin{enumerate}

\item {\tt STEPS}: This text field specifies the number of update
  steps of the network. With {\tt Topological\_Order} selected as
  update mode (chosen with the menu {\tt select update function} from
  the button {\tt OPTIONS} in the control panel) one step is sufficient
  to propagate information from input to output. With other update
  modes or with recursive networks, several steps might be needed.

\item \fbox{\tt STEP}: After clicking this button, the simulator
  kernel executes the number of steps specified in the text field {\tt
    STEPS}. If {\tt STEPS} is zero, the units are only redrawn. The
  update mode selected with the button \fbox{\tt MODE} is used (see
  chapter~\ref{Modes}). The first update step in the mode {\tt
    topological} takes longer than the following, because the net is
  sorted topologically first. Then all units are redrawn.

\item {\tt COUNT}: The text field next to the {\tt STEP} button
  displays the steps, executed so far.

\item \fbox{\tt INIT}: Initializes the network with values according
  to the function and parameters given in the last line of the panel.

\item \fbox{\tt RESET}: The counter is reset and the units are
  assigned their initial activation.

\item \fbox{\tt ERROR}: 
\index{Error Button} 
\index{Statistics, computation} 
  By pressing the error button in the control panel,
  SNNS will print out several statistics. The formulas were
  contributed by Warren Sarle from the SAS institute. Note that these
  criteria are for {\it linear} models; they can sometimes be applied
  directly to nonlinear models if the sample size is large. A
  recommended reference for linear model selection criteria is
  \cite{Jud80}.

  Notation:\\
  \begin{tabular}{rcl}
      $n$&=&Number of observations (sample size)\\ 
      $p$&=&Number of parameters, to be estimated (i.e. weights)\\ 
      $SSE$&=&The sum of squared errors \\ 
      $TSS$&=&The total sum of squares corrected for the mean \\ 
             &&for the dependent variable\\ 
  \end{tabular}

  {\bf Criteria for adequacy of the estimated model in the sample}\\ 
  Pearson's $R^2$, the proportion of variance, is explained or accounted
  by the model:\\ \hspace*{\fill} $R^2 := 1- \frac{SSE}{TSS}$ \hspace*{\fill} 

  {\bf Criteria for adequacy of the true model in the population}\\ 
  The mean square error \cite{Jud80} is defined as: $ MSE :=
  \frac{SSE}{n-p}$, the root mean square error as: $ RMSE := \sqrt{MSE}$.

  The $R^2_{adj}$, the $R^2$ \cite{Jud80} adjusted for degrees of
  freedom, is defined as:\\ \hspace*{\fill} $ R^2_{adj} := 1-
    \frac{n-1}{n-p}\cdot (1-R^2) $ \hspace*{\fill} 

  {\bf Criteria for adequacy of the estimated model in the population}\\
  Anemiya's prediction criterion \cite{Jud80} is similar to the
  $R^2_{adj}$:\\ \hspace*{\fill}$ PC := 1- \frac{n+p}{n-p}\cdot
    (1-R^2) $\hspace*{\fill}

  The estimated mean square error of prediction ($J_p$) assuming that
  the values of the regressors are fixed and that the model is correct
  is:\\ \hspace*{\fill} $ J_p:= (n+p)\cdot MSE/n $ \hspace*{\fill}

  The conservative mean square error in prediction \cite{Weh94} 
  is:\\ \hspace*{\fill}$ CMSEP:=\frac{SSE}{n-2p}$ \hspace*{\fill}

  The generalized cross validation ($GCV$) is given by Wahba
  \cite{Gol79} as:\\ \hspace*{\fill}$  GCV:=\frac{SSE\cdot n}{(n-p)^2}
  $\hspace*{\fill} 

  The estimated mean square error of prediction assuming that both
  independent and dependent variables are multivariate normal is
  defined as:\\ \hspace*{\fill}$ GMSEP:=\frac{MSE(n+1)(n-2)}{n(n-p-1)}
  $\hspace*{\fill} 

  Shibata's criterion $SHIBATA:=\frac{SSE(n+2p)}{n}$ can be found in
  \cite{Shi81}.

  Finally, there is Akaikes information criterion \cite{Jud80}:\\ 
  \hspace*{\fill}$ AIC:=n\cdot \ln{\frac{SSE}{n}} + 2p $
  \hspace*{\fill}\\ and the Schwarz's Bayesian criterion
  \cite{Jud80}:\\ \hspace*{\fill}$ SBC:=n\cdot \ln{\frac{SSE}{n}} + n
  \cdot \ln{p} $.\hspace*{\fill}

  Obviously, most of these selection criteria do only make sense, if
  $n>>p$.


\item \fbox{\tt INFO}: Information about the net is written to the
  text window.

\item {\tt CYCLES}: This text field specifies the number of learning
  cycles. It is mainly used in conjunction with the next two buttons.
  A cycle (also called an epoch sometimes) is a unit of training where
  all patterns of a pattern file are presented to the network once.

\item \fbox{\tt SINGLE}: The net is trained with a single pattern for
the number of training cycles defined in the field {\tt CYCLES}. The
text window reports the error of the network every {\tt CYCLES}$ / 10$
cycles, i.e.\ independent of the number of training cycles only
10 numbers are generated. (This prevents flooding the user with
network performance data and slowing down the training by file
I/O). 

\index{Error of the Network}
The error reported in the text window is the sum of the quadratic
differences between the teaching input and the real output over all
output units. The average error per output unit is given behind {\tt
ave}.

\item \fbox{\tt ALL}: The net is trained with all patterns for the
number of training cycles specified in the field {\tt CYCLES}. This is
the usual way to train networks from the graphical user interface.
Note, that if cycles has a value of, say, 100, the button {\tt ALL}
causes SNNS to train all patterns once (one cycle = one epoch) and
repeat this 100 times (NOT training each pattern 100 times in a
row and then applying the next pattern).

The error reported in the text window is the sum of the quadratic
difference between the teaching input and the real output over all
output units summed over the number of patterns presented. The average
error per output unit is given behind {\tt ave}.

\item \fbox{\tt STOP}: Stops the teaching cycle. After completion of
the current step or teaching cycle, the simulation is halted
immediately.

\item \fbox{\tt TEST}: With this button, the user can test the
behavior of the net with all patterns loaded. The activation values
of input and output units are copied into the net. (For output units
see also button \fbox{\tt SHOW}). Then the number of update steps
specified in {\tt STEPS} are executed.

\item \fbox{\tt SHUFFLE}: It is important for optimal learning that
the various patterns are presented in different order in the
different cycles. A random sequence of patterns is created
automatically, if {\tt SHUFFLE} is switched on.

\item \fbox{\tt OPTIONS}: Offers the following menu:
\index{Control Panel!Options}

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt jog weights}       & change all link weights randomly\\
{\tt edit f-types}      & edit/create f-types\\
{\tt edit sites}        & edit/create sites\\
\hline
\end{tabular}
\end{center}

If {\tt jog weights} is selected, a popup window appears to specify
the value range ({\tt low limit .. high limit}) of the random noise to be
added to all links in the network.

\item \fbox{\tt act}: With this button, the user specifies the changes
  to the activation values of the output units when a pattern is
  applied with \fbox{\tt TEST}. The following table gives the three
  possible alternatives:

  \begin{center}
  \begin{tabular}{|l|l|}
    \hline {\tt None} & The output units remain unchanged.\\ {\tt Out}
    & The output values are computed and set,\\ & activations remain
    unchanged.\\ {\tt Act} & The activation values are set.\\ \hline
  \end{tabular}
  \end{center}

  The label of this button always displays the item selected from the
  menu.

\item {\tt PATTERN}: This text field displays the current pattern
  number.

\item \fbox{\tt DELETE}: The pattern whose number is displayed in the
  text field {\tt PATTERN} is deleted from the pattern file.

\item \fbox{\tt MOD}: The pattern whose number is displayed in the
  text field {\tt PATTERN} is modified in place.

  The current activation of the input units and the current output
  values of output units of the network loaded make up the input and
  output pattern. These values might have been set with the network
  editor and the Info panel before.

\item \fbox{\tt NEW}: A new pattern is defined that is added behind
  existing patterns. Input and output values are defined as above.
  This button is disabled whenever the current pattern set has
  variable dimensions.

\item \fbox{\tt GOTO}: The simulator advances to the pattern whose
  number is displayed in the text field {\tt PATTERN}.

\item Arrow buttons \epsfbox{xgui_figs/xgui_button_first.ps},
  \epsfbox{xgui_figs/xgui_button_prev.ps},
  \epsfbox{xgui_figs/xgui_button_next.ps}, and
  \epsfbox{xgui_figs/xgui_button_last.ps}: With these buttons, the
  user can navigate through all patterns loaded, as well as jump
  directly to the first and last pattern. Unlike with the button
  \fbox{\tt TEST} no update steps are performed here.

\item \fbox{\tt SUB PAT}: Opens the panel for sub-pattern handling. The
  button is inactive when the current pattern set has no variable
  dimensions. The sub-pattern panel is described in section
  \ref{varPat}.

\item \fbox{\tt DEL SET}: Opens the menu of loaded pattern sets.  The
  pattern set of the selected entry is removed from main memory. The
  corresponding pattern file remains untouched. When the current
  pattern set is deleted, the last in the list becomes current. When
  the last remaining pattern set is deleted, the current pattern set
  becomes undefined and the menu shows the entry {\tt No Files}.

\item \fbox{\tt USE}: Also opens the menu of loaded pattern sets. The
  pattern set of the selected entry becomes the current set. All
  training, testing, and propagation actions refer always to the
  current pattern set. The name of the corresponding pattern file is
  displayed next to the button in the {\it Current Pattern Set} field.

\item {\it Current Pattern Set}: This field displays the name of the
  pattern set currently used for training. When no current pattern set
  is defined the entry "{\tt Training Pattern File ?}" is displayed.

\item {\tt VALID}: Gives the intervals in which the training process
  is to be interrupted by the computation of the error on the
  validation pattern set. A value of 0 inhibits validation. The
  validation error is printed on the shell window and plotted in the
  graph display.

\item \fbox{\tt USE}: Opens the menu of loaded pattern sets. The
  pattern set of the selected entry becomes the current validation
  set. The name of the corresponding pattern file is displayed next
  to the button in the {\it Validation Pattern Set} field.

\item {\it Validation Pattern Set}: This field displays the name of the
  pattern set currently used for validation. When no current pattern set
  is defined the entry "{\tt Validation Pattern File ?}" is displayed.

\item {\tt LEARN}: Up to five fields to specify the parameters of the
  learning function. The number required and their resp. meaning
  depend upon the learning function used. Only as many widgets as
  parameters needed will be displayed, i.e. all widgets visible need
  to filled in. A description of the learning functions that are
  already built in into SNNS is given in
  section~\ref{LearningParameters}.

\item \fbox{\tt SEL. FUNC}: in the LEARN row invokes a menu to select
a learning function (learning procedure). The following learning
functions are currently implemented:
\index{Control Panel!Learning Function}
\index{Learning!Function}

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt ART1}              & ART1 learning algorithm \\
{\tt ART2}              & ART2 learning algorithm \\
{\tt ARTMAP}            & ARTMAP learning algorithm\\
                        & (all ART models by Carpenter \& Grossberg)\\
{\tt BBPTT}             & Batch-Backpropagation for recurrent networks~~\\
{\tt BPTT}              & Backpropagation for recurrent networks\\
{\tt Backpercolation}   & Backpercolation 1 (Mark Jurik)\\
{\tt BackpropBatch}     & Backpropagation for batch training\\
{\tt BackpropMomentum}  & Backpropagation with momentum term\\
{\tt BackpropWeightDecay} & Backpropagation with Weight Decay \\
{\tt CC}                & Cascade correlation meta algorithm\\
{\tt Counterpropagation}& Counterpropagation (Robert Hecht-Nielsen)\\
{\tt Dynamic\_LVQ}      & LVQ algorithm with dynamic unit allocation\\
{\tt Hebbian}           & Hebbian learning rule\\
{\tt JE\_BP}            & Backpropagation for Jordan-Elman networks\\
{\tt JE\_BP\_Momentum}  & BackpropMomentum for Jordan-Elman networks\\
{\tt JE\_Quickprop}     & Quickprop for Jordan-Elman networks\\
{\tt JE\_Rprop}         & Rprop for Jordan-Elman networks\\
\hline 
\end{tabular}
\end{center}
\begin{center}
\begin{tabular}{|l|l|}
\hline 
{\tt Kohonen}           & Kohonen Self Organizing Maps\\
{\tt Monte-Carlo}       & Monte-Carlo learning\\
{\tt PruningFeedForward}& Pruning algorithms \\
{\tt QPTT}              & Quickprop for recurrent networks\\
{\tt Quickprop}         & Quickprop (Scott Fahlman)\\
{\tt RCC}               & Cascade Correlation for recurrent networks~~~~~~\\
{\tt RM\_delta}         & Rumelhart-McClelland's delta rule\\
{\tt RadialBasisLearning} & Radial Basis Functions\\
{\tt RBF-DDA}             & modified Radial Basis Functions\\
{\tt Rprop}               & Resilient Propagation learning\\
{\tt SimAnn\_SS\_error}   & Simulated Annealing with SSE computation\\
{\tt SimAnn\_WTA\_error}  & Simulated Annealing with WTA computation\\
{\tt SimAnn\_WWTA\_error} & Simulated Annealing with WWTA computation\\
{\tt Std\_Backpropagation}& ``vanilla'' Backpropagation\\
{\tt TimeDelayBackprop}   & Backpropagation for TDNNs (Alex Waibel)\\
\hline 
\end{tabular}
\end{center}

\item {\tt UPDATE}: Up to five fields to specify the parameters of the
  update function. The number required and their resp. meaning depend
  upon the update function used. Only as many widgets as parameters
  needed will be displayed, i.e. all widgets visible need to filled
  in.

\item \fbox{\tt SEL. FUNC}: in the UPDATE row invokes a menu to select
  an update function. A list of the update functions that
  are already built in into SNNS and their descriptions is given in
  section~\ref{UpdateFuncs}.

\item {\tt INIT}: Five fields to specify the parameters of the init
  function. The number required and their resp. meaning depend upon
  the init function used. Only as many widgets as parameters needed
  will be displayed, i.e. all widgets visible need to filled in.

\item \fbox{\tt SEL. FUNC}: in the INIT row invokes a menu to select
  an initialization function. See section~\ref{InitFuncs} for a list
  of the init functions available as well as their description.

\end{enumerate}


\subsection{Info Panel}
\label{UnitInfoPanel}
\index{Info Panel}
\index{Panel!Info Panel}


\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.65}{xgui_figs/FigUnitInfoPanel.ps}
  \end{minipage}
\caption{Info panel}
\label{FigUnitInfoPanel}
\end{center}
\end{figure}

\index{Source Unit}
\index{Target Unit}
The info panel displays all data of two units and the link between
them. The unit at the beginning of the link is called {\tt SOURCE},
the other {\tt TARGET}. One may run sequentially through all
connections or sites of the {\tt TARGET} unit with the arrow buttons
and look at the corresponding source units and vice versa.

This panel is also very important for editing, since some operations
refer to the displayed {\tt TARGET} unit or ({\tt
SOURCE}$\rightarrow${\tt TARGET}) link. A default unit can also be
created here, whose values (activation, bias, IO-type, subnet number,
layer numbers, activation function, and output function) are
copied into all selected units of the net. 

The source unit of a link can also be specified in a 2D display by
pressing the middle mouse button, the target unit by releasing it. To
select a link between two units the user presses the middle mouse
button on the source unit in a 2D display, moves the mouse to the
target unit while holding down the mouse button and releases it at the
target unit. Now the selected units and their link are displayed in
the info panel. If no link exists between two units selected in a 2D
display, the {\tt TARGET} is displayed with its first link, thereby
changing {\tt SOURCE}.

In table~\ref{InfoItems} the various fields are listed. The fields in
the second line of the SOURCE or TARGET unit display the name of the
activation function, name of the output function, name of the f-type
(if available). The fields in the line {\tt LINK} have the following
meaning: weight, site value, site function, name of the site. Most
often only a link weight is available. In this case no information
about sites is displayed. 


\begin{table}[hbtp]
\begin{center}
\begin{tabular}{|l|l|l|c|l|}
\hline
Name    & Meaning    & Type  & set by & value range\\ 
\hline
no.     & unit no.   & Label &        & $1$..$2^{31}$\\
subn.   & subnet no. & Label &        & $-32736$..$32735$\\
io      & IO-type    &Label &\fbox{\tt OPTIONS} & I(nput), O(utput), H(idden),\\
        &            &       &                    & D(ual), S(pecial)\\
act.    &activation  & Text  & input  & float;\\
iact.   &initial act.& Text  & input  & float;\\
out.    &output      & Text  & input  & float;\\
bias    &bias value  & Text  & input  & float\\
name    &unit name   & Text  & input  & string, starting with letter\\
\multicolumn{2}{|l|}{activation function}&Label&\fbox{\tt OPTIONS}& as available\\
\multicolumn{2}{|l|}{output function}& Label & \fbox{\tt OPTIONS} & as available\\
link    &weight      & Text  & input  & float\\
\multicolumn{2}{|l|}{site value}  & Label &        & float\\
\multicolumn{2}{|l|}{site function}& Label &        & as available\\
\multicolumn{2}{|l|}{site name} & Label &        & as available at {\tt TARGET}\\
\hline
\end{tabular}
\end{center}
\caption{Table of the unit, link and site fields in the Info panel}
\label{InfoItems}
\end{table}

Unit number, unit subnet number, site value, and site function cannot
be modified. To change attributes of type text, the cursor has to be
exactly in the corresponding field.

%\clearpage

There are the following buttons for the units (from left to right):

\begin{enumerate}

\item Arrow button \epsfbox{xgui_figs/xgui_button_first.ps}: The
  button below TARGET selects the first target unit (of the given
  source unit); the button below SOURCE selects the first source unit
  (of the given target unit);

\item Arrow button \epsfbox{xgui_figs/xgui_button_next.ps}: The
  button below TARGET selects the next target unit (of the given
  source unit); the button below SOURCE selects the next source unit
  (of the given target unit);

\item \fbox{\tt FREEZE}: Unit is frozen, if this button is inverted.
  Changes become active only after \fbox{\tt SET} is clicked.

\item \fbox{\tt DEF}: The default unit is assigned the displayed
  values of {\tt TARGET} and {\tt SOURCE} (only activation, bias,
  IO-type, subnet number, layer numbers, activation function and
  output function).

\item \fbox{\tt OPTIONS}: Calls the following menu:
%\clearpage

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt change io-type}    & change the IO-type\\
{\tt change f-type}     & change f-type\\
{\tt display activation function} 
                        &  graph of the activation function\\
{\tt change activation function} 
                        &  change activation function\\
                        & note: f-type gets lost!\\
{\tt display output function} 
                        &  graph of the output function\\
{\tt change output function} 
                        & change output function\\
                        & note: f-type gets lost!\\
{\tt assign layers}     & assign unit to layers\\
{\tt list all sources}  & list all predecessors\\
{\tt list all targets}  & list all successors\\
\hline
\end{tabular}
\end{center}

\item \fbox{\tt SET}: Only after clicking this button the attributes
of the corresponding unit are set to the specified value. The unit is
also redrawn. Therefore the values can be changed without immediate
effect on the unit.
\end{enumerate}

\bigskip
There exist the following buttons for links (from left to right):

\begin{enumerate}
\item \hspace{0.2cm} \epsfbox{xgui_figs/xgui_button_first.ps}: Select
first link of the {\tt TARGET} unit.

\item \hspace{0.2cm} \epsfbox{xgui_figs/xgui_button_next.ps}: Select
next link of the {\tt TARGET} unit.

\item \fbox{\tt OPTIONS}: Calls the following menu:

\begin{center}
\begin{tabular}{|l|l|}
\hline
{\tt list current site of TARGET} 
                        & list of all links of current site.\\
{\tt list all sites of TARGET} 
                        & list all sites of the {\tt TARGET}\\
{\tt list all links from SOURCE} 
                        & list all links starting at {\tt SOURCE}\\
{\tt delete site}       & delete displayed site\\
                        & note: f-type gets lost!\\
{\tt add site}          & add new site to {\tt TARGET}\\
                        & note: f-type gets lost!\\
\hline
\end{tabular}
\end{center}

\item \fbox{\tt SET}: Only after clicking this button the link weight
is set.
\end{enumerate}    


\subsubsection{Unit Function Displays}
\index{Unit!Function Display}
\index{Display!Unit Function Display}

The characteristic functions of the units can be displayed in a graphic
representation. For this purpose separate displays have been created,
that can be called by selecting the options {\tt display activation
function} or {\tt display output function} in the menu under the {\tt
options} button of the target and source unit in the info panel.
 
\begin{figure}[hbt]
\begin{center}
  \begin{minipage}[hbt]{\textwidth}
    \screendump{0.4}{xgui_figs/actfunc2.ps}
  \end{minipage}
\caption{\label{act_func2} The logistic activation function in a unit
  function display}
\end{center}
\end{figure}

Figure~\ref{act_func2} shows an example of an activation function. The
window header states whether it is an activation or an output
function, as well as whether it is the current function of the source
or target unit.

The size of the window is as flexible as the picture range of the
displayed function. The picture range can be changed by using the
dialog widgets at the top of the function displays. The size of the
window may be changed by using the standard mechanisms of your window
manager.

If a new activation or output function has been defined for the unit,
the display window changes automatically to reflect the new situation.
Thereby it is easy to get a quick overview of the available functions
by opening the function displays and then clicking through the list of
available functions (This list can be obtained by selecting {\tt
  select activation function} or {\tt select output function} in the
unit menu).


\subsection{2D Displays}
\label{GraphicWindows}
\index{2D Display}
\index{Display!2D Display}

\index{2D Display!Unit Attribute} A 2D display or simply display is
always part of the user interface. It serves to display the network
topology, the units' activations and the weights of the links. Each
unit is located on a grid position, which simplifies the positioning
of the units. The distance between two grid points ({\tt grid width})
can be changed from the default 37 pixels to other values in the setup
panel.

The current position, i.e.\ the grid position of the mouse, is also
numerically displayed at the bottom of the manager panel. The x-axis
is the horizontal line and valid coordinates lie in the range $-32736$
\ldots $+32735$ (short integer).

The current version displays units as boxes, where the size of the box
is proportional to the value of the displayed attribute. Possible
attributes are activation, initial activation, bias, and output. A
black box represents a positive value, an empty box a negative value.
The size of the unit varies between 16x16 and 0 pixels according to
the value of $scaleFactor$. The parameter {\em scaleFactor} has a
default value of $1.0$, but may be set to values between $0.0$ and
$2.0$ in the setup panel. Each unit can be displayed with two of
several attributes. One above the unit and one below the unit. The
attributes to be displayed can be selected in the setup panel.

Links are shown as solid lines, with optional numerical display of the
weight in the center of the line and/or arrow head pointing to the
target unit. These features are optional, because they heavily affect
the drawing speed of the display window.

A display can also be frozen with the button \fbox{\tt FREEZE} (button
gets inverted). It is afterwards neither updated anymore\footnote{If a
frozen display has to be redrawn, e.g.\ because an overlapping window
was moved, it gets updated. If the network has changed since the
freeze, its contents will also have changed!}, nor does it accept
further editor commands.

An iconified display is not updated and therefore consumes (almost) no
CPU time. If a window is closed, its dimensions and setup parameters
are saved in a stack (LIFO). This means that a newly requested display
gets the values of the window assigned that was last closed. For
better orientation, the window title contains the subnet number which
was specified for this display in the setup panel.


\subsubsection{Setup Panel of a 2D Display}
\label{SetupPanel}
\index{Setup Panel}
\index{Panel!Setup Panel}

Changes to the kind of display of the network can be performed in the
{\bf Setup panel}. All settings become valid only after the button
\fbox{\tt DONE} is clicked. The whole display window is then redrawn.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/FigSetupPanel.ps}
  \end{minipage}
\caption{Setup Panel of a 2D-display.}
\label{FigSetupPanel}
\end{center}
\end{figure}

\begin{enumerate}

\index{Display!2D Display!Unit Information}
\item Buttons to control the display of unit information: The first
two lines of the Setup panel ({\tt units top} and {\tt units bottom})
contain two buttons each to set the unit parameter that can be
displayed at the top resp.\ the bottom of the unit.

The button \fbox{\tt ON} toggles the display of information which can
be selected with the button \fbox{\tt SHOW}.  The unit name, unit
number, or the z-value (3D coordinate) can be displayed above the unit,
the activation, initial activation, bias, or output of the unit below
the unit. The numerical attribute selected with the button \fbox{\tt
SHOW} at the bottom of the unit (activation, initial activation,
output, or bias) also determines the size of the unit in the graphical
representation.

It is usually not advisable to switch off {\tt top} (number or name),
because this information is needed for reference to the info panel. An
unnamed unit is always displayed with its number.

\index{Display!2D Display!Link Information}
\item Buttons to control the display of link information: The third
line consists of three buttons to select the display of link data, 
\fbox{\tt ON}, \fbox{\tt $-2.35$}, \fbox{$\rightarrow$}.

\begin{itemize}
\item \fbox{\tt ON} determines whether to draw links at all (then
        \fbox{\tt ON} is inverted),
\item \fbox{\tt $-2.35$} displays link weights at the center of the
        line representing the link,
\item \fbox{$\rightarrow$} displays arrow heads of the links pointing
        from source to target unit.
\end{itemize}

\index{Display!2D Display!Layer}
\item \fbox{\tt LAYERS} invokes another popup window to select the
display of up to eight different layers in the display window. Layers are
being stacked like transparent sheets of paper and allow for a selective
display of units and links. These layers need NOT
correspond with layers of units of the network topology (as in
multilayer feed-forward networks), but they may do so. Layers are very
useful to display only a selected subset of the network. The display
of each layer can be switched on or off independently. A unit may
belong to several layers at the same time. The assignment of units to
layers can be done with the menu {\tt assign layers} invoked with the
button {\tt OPTIONS} in the main Info panel.

\index{Display!2D Display!Color}
\item \fbox{\tt COLOR} sets the 2D--display colors. On monochrome
  terminals, black on white or white on black representation of the
  network can be selected from a popup menu. On color displays, a
  color editing window is opened. This window consists of three parts:
  The palette of available colors at the top, the buttons to select
  the item to be colored in the lower left region, and the color
  preview window in the lower right region.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.5}{xgui_figs/colorSetup.ps}
  \end{minipage}
\caption{Color Setup Panel.}
\end{center}
\end{figure}

A color is set by clicking first at the appropriate button (\fbox{\tt
TEXT}, \fbox{\tt BACKGROUND}, or \fbox{\tt SELECTION}) and then at the
desired color in the color palette. The selected setting is immediately
displayed in the color preview window. All colors may be set in
any order and any number of times. The changes become effective in the
corresponding 2D--display only after both the setup panel and the
color edit panel have been dismissed with the \fbox{\tt DONE} button.

\index{Display!2D Display!Link Parameter}
\item Sliders for the selection of link display parameters,
{\tt links positive} and {\tt links negative}:

There are two slidebars to set thresholds for the display of links.
When the bubble is moved, the current threshold is displayed in
absolute and relative value at the bottom of the setup panel. Only
those links with an absolute value above the threshold are displayed.
The range of the absolute values is $0.0 \leq linkTrigger \leq 10.0$
(see also paragraph~\ref{GraphicWindows}). The trigger values can be
set independently for positive and negative weights. With these link
thresholds the user can concentrate on the strong connections.
Reducing the number of links drawn is an effective means to speed up
the drawing of the displays, since line drawing takes most of the time
to display a network.

Note: The links that are not drawn are only invisible. They still
remain accessible, i.e.\ they are affected by editor operations.

\index{Unit!Scale}
\item {\tt units scale}: This slidebar sets the parameter {\sl scaleFactor}
for the size of the growing boxes of the units. Its range is $0.0 \leq
scaleFaktor \leq 2.0$. A scale factor of $0.5$ draws the unit with
activation $0.5$ with full size. A scale factor of $2.0$ draws a unit
with activation $1.0$ only with half size.

\index{Grid!Width}
\item {\tt grid width}: This value sets the width of the grid on which
the units are placed. For some nets, changing the default of 37 pixels
may be useful, e.g.\ to be able to better position the units in a
geometrical pattern. Overlapping tops and bottoms occur if a grid
size of less than 35 pixels is selected (26 pixels if units are
displayed without numerical values). This overlap, however, does not
affect computation in any way.

\index{Grid!Origin}
\item {\tt origin (grid)}: 
These two fields determine the origin of the window, i.e.\ the grid
position of the top left corner.  There, the left field
represents the x coordinate, the right is the y coordinate. The origin
is usually (0, 0). Setting it to (20, 0) moves the display 20 units to
the right and 10 units down in the grid.

\index{Subnet Number}
\item {\tt subnet number}: 
This field adjusts the subnet number to be displayed in this window.
Values between $-32736$ and $+32735$ are possible here.
\end{enumerate}

%\clearpage

\subsection{Graph Window}
\index{Graph Window} 
\index{Window!Graph Window} 
\index{Error Curve}
Graph is a tool to visualize the error development of a net. The
program is started by clicking the \fbox{\tt graph} button in the
manager panel or by typing Alt-g in any SNNS window.
Figure~\ref{errorGraph1} shows the window of the graph tool.


\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/errorGraph1.ps}
  \end{minipage}
\caption{Graph window}
\label{errorGraph1}
\end{center}
\end{figure}

Graph is only active after calling it. This means, the development of
the error is only drawn as long as the window is not closed. The
advantage of this implementation is, that the simulator is not slowed
down as long as graph is closed\footnote{The loss of power by graph
should be minimal.}. If the window is iconified, graph remains
active.

The error curve of the net is plotted until the net is initialized or
a new net is loaded, in which case the cycle counter is reset to
zero. The window, however, is not cleared until the clear button is
pressed. This opens the possibility to compare several error curves in
a single display (see also figure~\ref{errorGraph1}). The maximum number of
curves, which can be displayed simultaneously is 25. If a $26^{th}$ curve is
tried to be drawn, the confirmer appears with an error message.

When the curve reaches the right end of the window, an automatic
rescale of the x-axis is performed. This way, the whole curve always
remains visible.

In the top region of the graph window, seven buttons for handling the
display are located:

\fbox{\tt CLEAR}: Clears the screen of the graph window and sets the cycle
counter to zero.

\fbox{\tt DONE}: Closes the graph window and resets the cycle counter.

For both the x-- and y--axis the following two buttons are available:
 
\hspace{0.0cm} \epsfbox{xgui_figs/xgui_button_prev.ps}: Reduce scale in one
direction.

\hspace*{0.0cm} \epsfbox{xgui_figs/xgui_button_next.ps}: Enlarge scale in one
direction.

\fbox{\tt SSE}: Opens a popup menu to select the value to be
plotted. Choices are \fbox{\tt SSE}, \fbox{\tt MSE}, and \fbox{\tt SSE/out}, the
SSE divided by the number of output units.

While the simulator is working all buttons are blocked.

The graph window can be resized by the mouse like every X-window. Changing
the size of the window does not change the size of the scale. 

When validation is turned on in the control panel two curves will be
drawn simultaneously in the graph window, one for the training set and
one for the validation set. On color terminals the validation error will
be plotted as solid red line, on B/W terminals as dashed black line.


\subsection{Weight Display}
\index{Weight!Display}
\index{Display!Weight Display}

\begin{figure}[hbt]
\begin{center}
  \begin{minipage}[hbt]{\textwidth}
    \screendump{0.7}{xgui_figs/hinton.ps2}
  \end{minipage}
  \caption{\label{Hinton} A typical Hinton diagram}
\end{center}
\end{figure}

The weight display window is a separate window specialized for
displaying the weights of a network. It is called from the manager
panel with the \fbox{\tt WEIGHTS} button, or by typing Alt-w in any
snns window. On black-and-white screens the weights are represented as
squares with changing size in a Hinton diagram, while on color
screens, fixed size squares with changing colors (WV-diagrams) are
used. It can be used to analyze the weight distribution, or to observe
the weight development during learning.

Initially the window has a size of 400x400 pixel.The weights are
represented by $16^2$ pixels on B/W and $5^2$ pixels on color terminals.
If the net is small, the square sizes are automatically enlarged to fill up
the window. If the weights do not fit into the window, the scrollbars
attached to the window allow scrolling over the display.

These settings may be changed by the user by pressing the
\fbox{ZOOM IN} and \fbox{ZOOM OUT} buttons in the upper part of the
window. \fbox{ZOOM IN} enlarges the weight square by one pixel on each
side, while \fbox{ZOOM OUT} shrinks it.

The setup panel lets the user change the look of the display
further. Here the width of the underlying grid can be changed. If the
grid size is bigger than the number of connections in the network, no
grid will be displayed. Also the color scale (resp. size scale for
B/W) can be changed here. The initial settings correspond to the SNNS
variables {\tt max\_weight} and {\tt min\_weight}.


\index{Hinton Diagram}
In a Hinton diagram, the size of a square corresponds to the absolute
size of the correlated link. A filled square represents positive, an
square frame negative links. The maximum size of the squares is
computed automatically, to allow an optimal use of the display.
\index{WV Diagram}
In a WV diagram color is used to code the value of a link. Here, a bright
red is used for large negative values and a bright green is used for
positive values. Intermediate numbers have a lighter color and the
value zero is represented by white. A reference color scale is
displayed in the top part of the window. The user also has the
possibility to display the numerical value of the link by clicking
any mouse button while the mouse pointer is on the square. A popup
window then gives source and target unit of the current link as well
as its weight.

For a better overall orientation the numbers of the units are printed
all around the display and a grid with user definable size is used. In
this numbering the units on top of the screen represent source units,
while numbers to the left and right represent target units.


\subsection{Projection Panel}
\index{Projection Panel}
\index{Window!Projection Panel}

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{10cm}
  \screendump{0.5}{xgui_figs/ProjectPanel.ps}
  \end{minipage}
  \hfill
\parbox{5cm}{\caption{PROJECTION panel for the 2-spirals problem.
    Note that the input range of the X and Y units must be specified,
    not the output range. In contrast, the output range of the third
    unit specified determines the color. Here the following values
    have been used: X-axis: unit 1 value range -6.5 to 6.5; Y-axis:
    unit 2 value range -6.5 to 6.5; activation pattern: unit 38 (the
    only output unit of the network) value range 0 to 1
    \label{ProjectionPanel}}}
\end{center}
\end{figure}

The projection analysis tool allows to display how the output of one
unit (e.g. a hidden or an output unit) depends on two input
units. It thus realizes a projection to two input vector axes.

It can be called by clicking the \fbox{\tt PROJECTION} button in the
manager panel or by typing Alt-p in any SNNS window. The display of the
projection panel is similar to the weights display, from which it is
derived.

in the setup panel, two units must be specified, whose inputs are
varied over the given input value range to give the X resp. Y
coordinate of the projection display. The third unit to be specified
is the one whose output value determines the color of the points with
the given X and Y coordinate values. The range for the color
coding can be specified as output range. For the most common logistic
activation function this range is $[0, 1]$.

The use of the other buttons, \fbox{ZOOM IN}, \fbox{ZOOM OUT} and
\fbox{DONE} are analogous to the weight display and should be obvious.

The projection tool is very instructive with the 2-spirals problem,
the XOR problem or similar problems with two-dimensional input. Each
hidden unit or output unit can be inspected and it can be determined,
to which part of the input space the neuron is sensitive. Comparing
different networks trained for such a problem by visualizing to which
part of the input space they are sensitive gives insights about the
internal representation of the networks and sometimes also about
characteristics of the training algorithms used for training. A
display of the projection panel is given in figure \ref{ProjectionPanel}.


\subsection{Print Panel}
\label{PrintPanel}
\index{Print Panel}
\index{Panel!Print Panel}

The print panel handles the Postscript output. A 2D-display can be
associated with the printer. All setup options and values of this
display will be printed. Color and encapsulated Postscript are also
supported. The output device is either a printer or a file.  If the
output device is a printer, a '.ps'-file is generated and spooled in the
/tmp directory. It has a unique name starting with the prefix `snns'. The
directory must be writable. When xgui terminates normally, all SNNS
spool files are deleted.


\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.7}{xgui_figs/printerSetup.ps}
  \end{minipage}
\caption{Printer panel}
\label{FigPrintPanel}
\end{center}
\end{figure}

The following fields can be set in the Printer Panel, which is shown
in figure~\ref{FigPrintPanel}.

\begin{enumerate}
\item{\bf File Name} resp. {\bf Command Line}:\\ 
If the output device is a file: the filename.\\
If the output device is a printer: the command line to start the printer.\\
The filename in the command line has to be '\$1'.
\item{\bf Destination}: Selects the output device.
Toggles the above input line between File Name and Command Line.
\item{\bf Paper}: Selects the paper format.
\item{\bf Orientation}: Sets the orientation of the display on the paper.
Can be 'portrait' or 'landscape'.
\item{\bf Border (mm)}: Sets the size of the horizontal and vertical
borders on the sheet in millimeters.
\item{\bf AutoScale}: Scales the network to the largest size possible
on the paper.
\item{\bf Aspect}: If on, scaling in X and Y direction is done uniformly.
\item{\bf X-Scale}: Scale factor in X direction. Valid only if AutoScale is 
'OFF'.
\item{\bf Y-Scale}: Scale factor in Y direction. Valid only if AutoScale is
'OFF'.
\end{enumerate}

\fbox{\tt DONE}: Cancels the printing and closes the panel.

\fbox{\tt PRINT}: Starts printing.

\fbox{\tt NETWORK}: Opens the network setup panel. This
panel allows the specification of several options to control the way
the network is printed.
\index{Network!Setup Panel}
\index{Panel!Network Setup Panel}

The variables that can be set here include:

\begin{enumerate}

\item{\bf x-min}, {\bf y-min}, {\bf x-max} and {\bf y-max} describe the
section to be printed. 
\item\begin{tabbing}{\bf Unit size}:\= \fbox{\tt FIXED}: All units have the same size.\\
\>\fbox{\tt VALUE}: The size of a unit depends on its value.
\end{tabbing}
\item{\bf Shape}: Sets the shape of the units.

\item\begin{tabbing}{\bf Text}:\= \fbox{\tt SOLID}: The box around
text overwrites the background color and the links.\\
\>\fbox{\tt TRANSPARENT}: No box around the text.
\end{tabbing}
\item{\bf Border}: A border is drawn around the network, if set to 'ON'.
\item{\bf Color}: If set, the value is printed color coded.
\item{\bf Fill Intens}: The fill intensity for units on monochrome
printers. 
\item{\bf Display}: Selects the display to be printed.
\end{enumerate}


\subsection{Help Windows}
\label{HelpWindows}
\index{Help!Window}
\index{Window!Help Window}

An arbitrary number of help windows may be opened, each displaying a
different part of the text. For a display of context sensitive help
about the editor commands, the mouse must be in a display and the key
\fbox{\tt h} must be pressed. Then the last open help window appears
with a short description.

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
    \screendump{0.5}{xgui_figs/FigHelpWindow.ps}
  \end{minipage}
\caption{Help Window}
\label{FigHelpWindow}
\end{center}
\end{figure}


A special feature is the possibility of searching a given string in the
help text. For this, the search string is selected in the text window
(e.g.\ by a double click).

\begin{enumerate}

\item \fbox{\tt LOOK}: After clicking this button, SNNS looks for the
first appearance of the marked string, starting at the beginning of
the help document. If the string is found, the corresponding paragraph
is displayed.

\item \fbox{\tt MORE}: After clicking this button, SNNS looks for the
first appearance of the marked string, starting at the position last
visited by a call to the help function. If the text was scrolled
afterwards, this position might not be on the display anymore.
\end{enumerate}

\index{Help.hdoc!Modification}
Note: All help calls look for the first appearance of a certain
string. These strings start with the sequence ASTERISK-BLANK (\verb*@*
@), to assure the discovery of the appropriate text position. With
this knowledge it is easy to modify the file {\tt help.hdoc} to adapt
it to special demands, like storing information about unit types or
patterns. The best approach would be to list all relevant keywords at
the end of the file under the headline ``{\tt * TOPICS}'', so that the
user can select this directory by a click to \fbox{\tt TOPICS}.


\subsection{Shell window}
The window of the shell from which SNNS is invoked is used for the
output of protocol messages. 

These protocols include:
\begin{itemize}
\item Messages about the success or failure of the loading or saving
of a file.

\item Information about the settings of SNNS when the \fbox{INFO}
button in the control panel is pressed.

\item Error messages of the pattern file parser when the pattern file
does not correspond to the required grammar.

\item Learning error values (see below)

\item Validation set error values
\end{itemize}

When learning is started, the error of the output units is reported on
this window after each epoch, i.e. after the presentation of all
patterns.

      To save the window from being flooded on longer training runs, the
maximum number of reported errors is limited to 10. Therefore, when 20
learning cycles are specified, the error gets printed only after every
other cycle. This error report has the following form:

\small
\begin{verbatim}
        Learning all patterns:
          epochs   : 100
          parameter:  0.80000
          #o-units : 26
          #patterns: 26

           epoch:         SSE            MSE            SSE/o-units

      Train  100:      57.78724        2.22259           2.22259
      Train   90:      24.67467        0.94903           0.94903
      Train   80:      23.73399        0.91285           0.91285
      Train   70:      22.40005        0.86154           0.86154
      Train   60:      20.42843        0.78571           0.78571
      Train   50:      18.30172        0.70391           0.70391
      Test    50:      25.34673        0.97487           0.97487
      Train   40:      16.57888        0.63765           0.63765
      Train   30:      14.84296        0.57088           0.57088
      Train   20:      12.97301        0.49896           0.49896
      Train   10:      11.22209        0.43162           0.43162
      Train    1:      10.03500        0.38596           0.38596
      Test     1:      11.13500        0.42696           0.42696
\end{verbatim}
\normalsize

The first line reports whether all or only a single pattern is
trained. The next lines give the number of specified cycles and the
given learning parameters, followed by a brief setup description.

Then the 10-row-table of the learning progress is given. If validation
is turned on this table is intermixed with the output of the
validation. The first column specifies whether the displayed error is
computed on the training or validation pattern set, ``Test'' is
printed for the latter case. The second column gives the number of
epochs still to be processed. The third column is the Sum Squared
Error (SSE) of the learning function. It is computed with the
following formula:

\[ SSE = \sum_{p \in patterns} \sum_{j \in output} (t_{pj} - o_{pj})^2 \]

where $t_{pj}$ is the teaching output (desired output) of output
neuron j on pattern p and $o_{pj}$ is the actual output.  The forth
column is the Mean Squared Error (MSE), which is the SSE divided by
the number of patterns.  The fifth value finally gives the SSE divided
by the number of output units.

The second and third values are equal if there are as many patterns as
there are output units (e.g.  the letters network), the first and
third values are identical, if the network has only one output unit
(e.g.~the xor network).

If the training of the network is interrupted by pressing the
\fbox{STOP} button in the control panel, the values for the last
completed training cycle are reported. 


\subsection{Confirmer}
\label{Confirmer}
\index{Confirmer}

\begin{figure}[htbp]
\begin{center}
  \begin{minipage}{\textwidth}
  \begin{minipage}{8cm}
    \centerline{\screendump{0.5}{xgui_figs/FigConfirmer_a.ps}}
    \hfill
  \end{minipage}
  \begin{minipage}{6cm}
    \vspace*{-4mm}
    \centerline{\screendump{0.5}{xgui_figs/FigConfirmer_b.ps}}
  \end{minipage}
  \end{minipage}
\caption{A normal confirmer and a message confirmer.}
\label{FigConfirmer}
\end{center}
\end{figure}

The  confirmer is a window where the graphical user interface
displays important information or requires the user to confirm
destructive operations. The confirmer always appears in the middle of
the screen and blocks XGUI until a button of the confirmer is
clicked (see figure~\ref{FigConfirmer}).

\clearpage

\section{Parameters of the Learning Functions}
\label{LearningParameters}
\index{Control Panel!Learning Function!Parameter}

The following learning parameters (from left to right) are used by the learning
functions that are already built into SNNS:

\begin{itemize}
\item {\tt Std\_Backpropagation} (''Vanilla`` Backpropagation),\\
{\tt BackpropBatch} and\\ {\tt TimeDelayBackprop}
\index{Learning!Function Parameter!BackpropBatch}
\index{BackpropBatch}
\index{Learning!Function Parameter!Std\_Backpropagation}
\index{Std\_Backpropagation}
\index{Learning!Function Parameter!TimeDelayBackprop}
\index{TimeDelayBackprop}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.
        
        Typical values of $\eta$ are $0.1 \ldots 1.0$. Some small examples
        actually train even faster with values above 1, like 2.0.

  \item $d_{max}$: the maximum difference 
        $d_{j} = t_{j} - o_{j}$ between a teaching value $t_{j}$ and 
        an output $o_{j}$ of an output unit which is tolerated, 
        i.e.\ which is propagated back as $d_{j} = 0$. 
        If values above 0.9 should be regarded as 1 and values 
        below 0.1 as 0, then $d_{max}$ should be set to $0.1$. 
        This prevents overtraining of the network.

        Typical values of $d_{max}$ are 0, 0.1 or 0.2.
  \end{enumerate}

%\clearpage

\item {\tt BackpropMomentum} (Backpropagation with momentum term and
        flat spot elimination):
\index{Learning!Function Parameter!BackpropMomentum}
\index{BackpropMomentum}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.

        Typical values of $\eta$ are $0.1 \ldots 1.0$. Some small examples
        actually train even faster with values above 1, like 2.0.

  \item $\mu$: momentum term, specifies the amount of the old weight
        change (relative to 1) which is added to the current change.

        Typical values of $\mu$ are $0 \ldots 1.0$.

  \item $c$: flat spot elimination value, a constant value 
        which is added to the derivative of the activation function
        to enable the network to pass flat spots of the error surface.

        Typical values of $c$ are $0 \ldots 0.25$, most often
        0.1 is used.

  \item $d_{max}$: the maximum difference 
        $d_{j} = t_{j} - o_{j}$ between a teaching value $t_{j}$ and 
        an output $o_{j}$ of an output unit which is tolerated, 
        i.e.\ which is propagated back as $d_{j} = 0$. See above.
  \end{enumerate}

\index{Backpropagation!General Formula}
  The general formula for Backpropagation used here is
  \begin{eqnarray*}
        \Delta w_{ij}(t+1) & = & \eta \  \delta_{j} \  o_{i} 
                                + \mu \ \Delta w_{ij}(t) \\
        \delta_{j}    & = & \left\{
        \begin{array}{ll}
        (f'_{j}(net_{j}) + c) (t_{j} - o_{j}) &
        \mbox{if unit $j$ is a output-unit}\\
        (f'_{j}(net_{j}) + c) \sum_{k} \delta_{k} w_{jk} & 
         \mbox{if unit $j$ is a hidden-unit}\\
        \end{array}
        \right. \\
  \end{eqnarray*}

\clearpage
\item{\tt BackpropWeightDecay} (Backpropagation with Weight Decay)
\index{Learning!Function Parameter!BackpropWeightDecay}
\index{BackpropWeightDecay}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.
        
        Typical values of $\eta$ are $0.1 \ldots 1.0$. Some small examples
        actually train even faster with values above 1, like 2.0.

  \item $d$: weight decay term, specifies how much of the old weight
        value is subtracted after learning. Try values between 0.005
        and 0.3.

  \item $d_{min}$: the minimum weight that is tolerated for a
    link. All links with a smaller weight will be pruned.

  \item $d_{max}$: the maximum difference 
        $d_{j} = t_{j} - o_{j}$ between a teaching value $t_{j}$ and 
        an output $o_{j}$ of an output unit which is tolerated, 
        i.e.\ which is propagated back as $d_{j} = 0$. See above.
  \end{enumerate}

%\clearpage 

\item {\tt BackpropThroughTime (BPTT)},\\
      {\tt BatchBackpropThroughTime (BBPTT)}:
\index{Learning!Function Parameter!BackpropThroughTime}
\index{BackpropThroughTime}
\index{Learning!Function Parameter!BatchBackpropThroughTime}
\index{BatchBackpropThroughTime}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.

        Typical values of $\eta$ for BPTT and BBPTT are $0.005 \ldots 0.1$.

  \item $\mu$:  momentum term, specifies the amount of the old weight
        change (relative to 1) which is added to the current change.
                
        Typical values of $\mu$ are $0.0 \ldots 1.0$.

  \item backstep: the number of backprop steps back in time.
        BPTT stores a sequence of all unit activations while input
        patterns are applied. The activations are stored in a
        first-in-first-out queue for each unit. The largest backstep 
        value supported is 10.
   \end{enumerate}

\item {\tt Quickprop}:
\index{Learning!Function Parameter!Quickprop}
\index{Quickprop}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.

        Typical values of $\eta$ for Quickprop are $0.1 \ldots 0.3$.

  \item $\mu$: maximum growth parameter, specifies the maximum amount
        of weight change (relative to 1) which is added to the current change

        Typical values of $\mu$ are $1.75 \ldots 2.25$.

  \item $\nu$: weight decay term to shrink the weights.

        Typical values of $\nu$ are $0.0001$. Quickprop is rather
        sensitive to this parameter. It should not be set too large.

  \item $d_{max}$: the maximum difference 
        $d_{j} = t_{j} - o_{j}$ between a teaching value $t_{j}$ and 
        an output $o_{j}$ of an output unit which is tolerated, 
        i.e.\ which is propagated back as $d_{j} = 0$. See above.
  \end{enumerate}

\clearpage 

\item {\tt QuickpropThroughTime (QPTT)}:
\index{Learning!Function Parameter!QuickpropThroughTime}
\index{QuickpropThroughTime}

  \begin{enumerate}
  \item $\eta$: learning parameter, specifies the step width of the
        gradient descent.

        Typical values of $\eta$ for QPTT are $0.005 \ldots 0.1$.

  \item $\mu$: maximum growth parameter, specifies the maximum amount
        of weight change (relative to 1) which is added to the current change

        Typical values of $\mu$ are $1.2 \ldots 1.75$.

  \item $\nu$: weight decay term to shrink the weights.

        Typical values of $\nu$ are $0.0005 \ldots 0.00005$. 

  \item backstep: the number of quickprop steps back in time.
        QPTT stores a sequence of all unit activations while input
        patterns are applied. The activations are stored in a
        first-in-first-out queue for each unit. 

        The largest backstep value supported is 10.
  \end{enumerate}

\item {\tt Counterpropagation}:
\index{Learning!Function Parameter!Counterpropagation}
\index{Counterpropagation}

  \begin{enumerate}
  \item $\alpha$: learning parameter of the Kohonen layer.

        Typical values of $\alpha$ for Counterpropagation are $0.1 \ldots
        0.7$ .

  \item $\beta$: learning parameter of the Grossberg layer.

        Typical values of $\beta$ are $0 \ldots 1.0$.

  \item $\theta$: threshold of a unit.

        We often use a value $\theta$ of 0.
  \end{enumerate}

\item {\tt Backpercolation 1}:
\index{Learning!Function Parameter!Backpercolation}
\index{Backpercolation}

  \begin{enumerate}
  \item $\lambda$: global error magnification. This is the factor in
        the formula $\epsilon = \lambda (t - o)$, where $\epsilon$ is
        the internal activation error of a unit, $t$ is the teaching
        input and $o$ the output of a unit.

        Typical values of $\lambda$ are 1. Bigger values (up to 10)
        may also be used here.

  \item $\theta$: If the error value $\xi$ drops below this threshold
        value, the adaption according to the Backpercolation algorithm
        begins. $\xi$ is defined as:

\[
\xi = \frac{1}{pN}\sum^p \sum^N \mid o - \phi \mid
\]

  \item $d_{max}$: the maximum difference 
        $d_{j} = t_{j} - o_{j}$ between a teaching value $t_{j}$ and 
        an output $o_{j}$ of an output unit which is tolerated, 
        i.e.\ which is propagated back as $d_{j} = 0$. See above.
  \end{enumerate}

\clearpage 

\item {\tt Dynamic Learning Vector Quantization (DLVQ)}:
\index{Learning!Function Parameter!DLVQ}
\index{DLVQ}
 
  \begin{enumerate}
  \item $\eta^{+}$: learning rate, specifies the step width of the mean
        vector $\vec{\mu_{A}}$, which is nearest to a pattern $\vec{x_{A}}$,
        towards this pattern. Remember that $\vec{\mu_{A}}$ is moved only, if
        $\vec{x_{A}}$ is not assigned to the correct class $w_{A}$. A typical
        value is 0.03.

  \item $\eta^{-}$: learning rate, specifies the step width of a mean
        vector $\vec{\mu_{B}}$, to which a pattern of class $w_{A}$ is falsely
        assigned to, away from this pattern. A typical value is 0.03. Best
        results can be achieved, if the condition $\eta^{+} = \eta^{-}$ is
        satisfied.

  \item Number of cycles you want to train the net before additive mean
        vectors are calculated.
  \end{enumerate}

\item {\tt RadialBasisLearning}:
\index{Learning!Function Parameter!RBF}
\index{RadialBasisLearning}

  \begin{enumerate}

  \item {\sl centers}: determines the learning rate $\eta_1$ used for the
        modification of center vectors.

  \item{\sl bias (p)}: determines the learning rate $\eta_2$, used for
        the modification of the parameters $p$ of the base function. $p$
        is stored as bias of the hidden units.

  \item{\sl weights}: influences the training of all link weights that
        are leading to the output layer as well as the training of the
        bias of all output neurons.

  \item{\sl delta max.}: If the actual error is smaller than the
        maximum allowed error ({\sl delta max.}) the corresponding
        weights are not changed.

  \item{\sl momentum}:influences the amount of the momentum--term during
        training.   

  \end{enumerate}
 
\item {\tt RadialBasisLearning with Dynamic Decay Adjustment}:
\index{Learning!Function Parameter!RBF-DDA}
\index{RBF-DDA}
\begin{enumerate}

\item $\theta^{+}$: positive threshold. To commit a new prototype,
  none of the existing RBFs of the correct class may have an
  activation above $\theta^{+}$

\item $\theta^{-}$:negative threshold. During shrinking no RBF unit of
  a conflicting class is allowed to have an activation above
  $\theta^{-}$.

\item n: the maximum number of RBF units to be diplayed in one row.
  This item allows the user to control the appearance of the network
  on the screen and has no influence on the performance.

\end{enumerate}

%\clearpage

\item {\tt ART1}
\index{Learning!Function Parameter!ART1}
\index{ART1}

  \begin{enumerate}

  \item $\rho$: vigilance parameter. If the quotient of active F$_1$ units
        divided by the number of active F$_0$ units is below $\rho$, an
        ART reset is performed.

  \end{enumerate}


\item {\tt ART2}
\index{Learning!Function Parameter!ART2}
\index{ART2}

  \begin{enumerate}

  \item $\rho$: vigilance parameter. Specifies the minimal length of the
        error vector {\bf r} (units $r_i$).
  \item $a$: Strength of the influence of the lower level in F$_1$ by
        the middle level.
  \item $b$: Strength of the influence of the middle level in F$_1$ by
        the upper level.
  \item $c$: Part of the length of vector {\bf p}
        (units  $p_i$) used to compute the error.
  \item $\Theta$: Threshold for output function $f$ of units $x_i$ and
        $q_i$.

  \end{enumerate}

%\clearpage

\item {\tt ARTMAP}
\index{Learning!Function Parameter!ARTMAP}
\index{ARTMAP}

  \begin{enumerate}

  \item $\overline{\rho^a}$: vigilance parameter for $ART^a$ subnet.
        (quotient $\frac{|\mbox{F}_1^a|}{|\mbox{F}_0^a|}$)
  \item $\rho^b$: vigilance parameter for $ART^b$ subnet.
        (quotient $\frac{|\mbox{F}_1^b|}{|\mbox{F}_0^b|}$)
  \item $\rho$: vigilance parameter for inter ART reset control. 
        (quotient $\frac{|\mbox{F}^{ab}|}{|\mbox{F}_2^b|}$)

  \end{enumerate}

\item{\tt RPROP (resilient propagation)}
\index{Learning!RPROP}
\index{RPROP}

  \begin{enumerate}

  \item {\sl delta$_0$}: starting values for all $\triangle_{ij}$.
        Default value is 0.1.

  \item{\sl $delta_{max}$}: the upper limit for the update values 
        $\triangle_{ij}$.The default value of $\triangle_{max}$ is $50.0$.

  \item{\sl $\alpha$}: the weight-decay determines the relationship
        between the output error and to reduction in the size of the
        weights.{\em Important:} Please note that the weight decay
        parameter $\alpha$ denotes the exponent, to allow comfortable
        input of very small weight-decay.  A choice of the third
        learning parameter $\alpha = 4$ corresponds to a ratio of 
        weight decay term to output error of $1:10000 (1:10^4)$.

  \end{enumerate}


\item {\tt Cascade Correlation (CC)} and\\
      {\tt Recurrent Cascade Correlation (RCC)}
\index{Learning!Function Parameter!CC}
\index{Learning!Function Parameter!RCC}

CC and RCC are not learning functions themselves. They are meta
algorithms to build and train optimal networks. However, they have a
set of standard learning functions embedded. Here these functions
require modified parameters. The embedded learning functions are:

\begin{itemize}         %%%%% itemize embedded functions

\item Backpropagation (in CC or RCC):  %%%%% embedded function
\begin{enumerate}
\item $\eta_{1}$: learning parameter, specifies the step width of gradient
      decent minimizing the net error.
\item $\mu_{1}$: momentum term, specifies the amount of the old weight
      change, which is added to the current change.
\item $c$: flat spot elimination value, a constant value which is added to
      the derivative of the activation function to enable the network
      to pass flat spots on the error surface.
\item $\eta_{2}$: learning parameter, specifies the step width of gradient
      ascent maximizing the covariance.
\item $\mu_{2}$: momentum term specifies the amount of the old weight change,
      which is added to the current change.                 
\end{enumerate}

The general formula for this learning function is: 
\begin{eqnarray*}
\Delta w_{ij}(t+1) & = & \eta S(t) + \mu \Delta w_{ij}(t-1)  
\end{eqnarray*}

The slopes $\partial E/\partial w_{ij}$ and $- \partial C/\partial w_{ij}$
are abbreviated by $S$. This abbreviation is valid for all embedded functions.
By changing the sign of the gradient value $ \partial C/\partial w_{ij}$, 
the same learning function can be used to maximize the covariance and to
minimize the error.

\item{Rprop} (in CC or RCC):            %%%%% embedded function
\begin{enumerate}
\item $\eta_{1}^{-}$: decreasing factor, specifies the factor by which
      the update-value $\Delta_{ij}$ is to be decreased when minimizing the
      net error. A typical value is $0.5$. 
\item $\eta_{1}^{+}$: increasing factor, specifies the factor by which
      the update-value $\Delta_{ij}$ is to be increased when minimizing the
      net error. A typical value is $1.2$
\item not used.
\item $\eta_{2}^{-}$: decreasing factor, specifies the factor by which
      the update-value $\Delta_{ij}$ is to be decreased when maximizing the
      covariance. A typical value is $0.5$. 
\item $\eta_{2}^{+}$: increasing factor, specifies the factor by which
      the update-value $\Delta_{ij}$ is to be increased when maximizing the
      covariance. A typical value is $1.2$
\end{enumerate}

The weight change is computed by:
\begin{eqnarray*}
\Delta w_{ij}(t) = \left\{
\begin{array} {ll}
- \Delta_{ij}(t-1) \eta^{-}, & \mbox{if } S(t) S(t-1) < 0 \\
- \Delta_{ij}(t-1) \eta^{+}, & \mbox{if } S(t)>0 \mbox{ and } S(t-1)>0 \\
\mbox{  } \Delta_{ij}(t-1) \eta^{+}, & \mbox{if } S(t)<0 \mbox{ and } S(t-1)<0 \\
\mbox{ } 0,                         & \mbox{else }                \\
\end{array} \right.
\end{eqnarray*}

where $\Delta_{ij}(t)$ is defined as follows: $\Delta_{ij}(t) =
\Delta_{ij}(t-1) \eta^{+/-}$.  Furthermore, the condition $0 <
\eta^{-} < 1 < \eta^{+}$ should not be violated.

\item{Quickprop} (in CC or RCC):        %%%%% embedded function
\begin{enumerate}
\item $\eta_{1}$: learning parameter, specifies the step width of the
      gradient descent when minimizing the net error. 
      A typical value is $0.0001$ 
\item $\mu_{1}$: maximum growth parameter, realizes a kind of dynamic 
       momentum term. A typical value is 2.0. 
\item $\nu$: weight decay term to shrink the weights. A typical value is
      $ <=0.0001$.
\item $\eta_{2}$: learning parameter, specifies the step width of the
      gradient ascent when maximizing the covariance. 
      A typical value is $0.0007$ 
\item $\mu_{2}$: maximum growth parameter, realizes a kind of dynamic 
       momentum term. A typical value is 2.0. 
\end{enumerate}

The formula used is:
\begin{displaymath}
\Delta w_{ij}(t) = \left\{
\begin{array}{ll}
\eta S(t), & \mbox{if } \Delta w_{ij}(t-1) = 0 \\
\frac{S(t)}{S(t-1)-S(t)} \Delta w_{ij}(t-1), & \mbox{if } \Delta 
w_{ij}(t-1) \not= 0 \mbox{ and } \frac{S(t)}{S(t-1)-S(t)} < \mu \\
\mu \Delta w_{ij}(t-1), & \mbox{else}
\end{array} \right.
\end{displaymath}
  
\end{itemize}           %%%%% itemize embedded functions

\clearpage

\item {\tt Kohonen}
\index{Learning!Function Parameter!Kohonen}
\index{Kohonen}
\index{SOM}

  \begin{enumerate}

  \item{\sl h(0):} Adaptation height. The initial adaptation height
  can vary between 0 and 1. It determines the overall adaptation
  strength.
 
  \item{\sl r(0):} Adaptation radius. The initial adaptation radius
  $r(0)$ is the radius of the neighborhood of the winning unit. All
  units within this radius are adapted. Values should range between 1
  and the size of the map.

  \item{\sl mult\_H:} Decrease factor. The adaptation height decreases
  monotonically after the presentation of every learning pattern. This
  decrease is controlled by the decrease factor mult\_H: $h(t+1) :=
  h(t) \cdot mult\_H$

  \item{\sl mult\_R:} Decrease factor. The adaptation radius also
  decreases monotonically after the presentation of every learning
  pattern. This second decrease is controlled by the decrease factor
  mult\_R: $r(t+1) := r(t) * mult\_R$

  \item{\sl h:} Horizontal size. Since the internal representation of a
  network doesn't allow to determine the 2-dimensional layout of the
  grid, the horizontal size in units must be provided for the learning
  function.  It is the same value as used for the creation of the
  network.  
  \end{enumerate}

\item {\tt RM\_delta (Rumelhart and McClelland's delta rule) }
\index{Learning!Function Parameter!RM\_delta}

  \begin{enumerate}
  
  \item{\sl n:} learning parameter, specifies the step width of the
      gradient descent. In \cite{rum861} Rumelhart and McClelland 
      use 0.01, although values less than 0.03 are generally
      acceptable.

  \item{\sl Ncycles:} number of update cycles, specifies how many
      times a pattern is propagated through the network before the
      learning rule is applied. This parameter must be large enough so
      that the network is relatively stable after the set number of
      propagations.  A value of 50 is recommended as a baseline.
      Increasing the value of this parameter increases the accuracy of
      the network but at a cost of processing time.  Larger networks
      will probably require a higher setting of Ncycles.

  \end{enumerate}
  {\bf NOTE:} With this learning rule the update function {\tt
    RM\_Synchronous} has to be used which needs as update parameter
  the number of iterations!

%\clearpage

\item {\tt Hebbian Learning}
\index{Learning!Function Parameter!Hebbian Learning}

  \begin{enumerate}
  
  \item{\sl n:} learning parameter, specifies the step width of the
      gradient descent. Values less than (1 / number of nodes) are
      recommended.

  \item{\sl Wmax:} maximum weight strength, specifies the maximum
      absolute value of weight allowed in the network.  A value of 1.0
      is recommended, although this should be lowered if the network
      experiences explosive growth in the weights and activations.
      Larger networks will require lower values of Wmax.
        
  \item{\sl count:} number of times the network is updated before 
      calculating the error.

  \end{enumerate}

  {\bf NOTE:} With this learning rule the update function {\tt
  RM\_Synchronous} has to be used which needs as update parameter
  the number of iterations!  

\item{\tt Monte-Carlo:} 
\index{Learning!FunctionParameter!Monte-Carlo} 
\index{Monte-Carlo}

  \begin{enumerate}

  \item {\sl $Min$}: lower limit of weights and biases.
    Typical values are $ -10.0 \ldots -1.0$.

  \item{\sl $Max$}: upper limit of weights and biases.
    Typical values are $ 1.0 \ldots 10.0$.

  \end{enumerate}

\item {\tt Simulated\_Annealing\_SS\_error},\\ 
{\tt Simulated\_Annealing\_WTA\_error} and\\ 
{\tt Simulated\_Annealing\_WWTA\_error:} 
\index{Learning!Function Parameter! Simulated\_Annealing\_SS\_error}
\index{Simulated\_Annealing\_SS\_error} 
\index{Learning!Function Parameter!Simulated\_Annealing\_WTA\_error}
\index{Simulated\_Annealing\_WTA\_error} 
\index{Learning!Function Parameter!Simulated\_Annealing\_WWTA\_error}
\index{Simulated\_Annealing\_WWTA\_error}

  \begin{enumerate}

  \item {\sl $Min$}: lower limit of weights and biases.
    Typical values are $-10.0 \ldots -1.0$.

  \item{\sl $Max$}: upper limit of weights and biases.
    Typical values are $ 1.0 \ldots 10.0$.

  \item $T_0$: learning parameter, specifies the Simulated Annealing
    start temperature .
    Typical values of $T_0$ are $1.0 \ldots 10.0$.

  \item $deg$: degradation term of the temperature: $T_{new} = T_{old}
    \cdot deg$
    Typical values of $deg$ are $0.99 \ldots 0.99999$.

  \end{enumerate}


\item {\tt Scaled Conjugate Gradient (SCG)}
\index{Scaled Conjugate Gradient}
\index{SCG}
 
All of the following parameters are non-critical, i.e. they influence
only the speed of convergence, not whether there will be success or
not.

   \begin{enumerate}
   \item $\sigma_1$. Should satisfy $0<\sigma_1\leq 10^{-4}$. If 0, will
     be set to $10^{-4}$;
   \item $\lambda_1$. Should satisfy $0<\lambda_1\leq 10^{-6}$. If 0,
     will be set to $10^{-6}$;
   \item $\Delta_{max}$. See standard backpropagation. Can be set to 0 if
     you don't know what to do with it;
   \item $\epsilon_1$. Depends on the floating-point precision. Should be set to
     $10^{-8}$ (simple precision) or to $10^{-16}$ (double precision). If
     0, will be set to $10^{-8}$.
\end{enumerate}

\end{itemize}           %%%%% itemize all learning functions



\section {Update Functions}
\label{UpdateFuncs}
\index{Update Functions}

Why is an update mode important? It is necessary to visit the neurons 
of a net in a specific sequential order to perform operations on them. 
This order depends on the topology of the net and greatly influences 
the outcome of a propagation cycle. For each net 
with its own characteristics, it is very important to choose the update 
function associated with the net in order to get the desired behavior 
of the neural network. If a wrong update function is given, SNNS will display 
 a error message on your screen. Click \fbox {\tt OPTIONS} in the control panel 
to select an update function.  
 
The following update functions are available for the various network types:

\begin{center}
\begin{tabular}{|l|l|}
\hline
ART1\_stable \hspace{0.5cm} & \hspace{0.5cm} for ART1 networks\\ 
ART1\_synchronous \hspace{0.5cm} & \hspace{0.5cm} for ART1 networks\\
ART2\_Stable \hspace{0.5cm} & \hspace{0.5cm} for ART2 networks\\
ART2\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for ART2 networks\\
ARTMAP\_Stable  \hspace{0.5cm} & \hspace{0.5cm} for ARTMAP networks\\
ARTMAP\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for ARTMAP networks\\
Auto\_Synchronous  \hspace{0.5cm} & \hspace{0.5cm} for Autoassociative Memory networks\\
BAM\_Order \hspace{0.5cm} & \hspace{0.5cm} for Bidirectional Associative Memory networks\\
BBTT\_Order \hspace{0.5cm} & \hspace{0.5cm} for Backpropagation-Through-Time networks\\
CC\_Order \hspace{0.5cm} & \hspace{0.5cm} for Cascade- Correlation networks\\
CounterPropagation \hspace{0.5cm} & \hspace{0.5cm} for Counterpropagation networks\\
Dynamic\_LVQ \hspace{0.5cm} & \hspace{0.5cm} for Dynamic-Learning-Vector-Quantization networks\\
Hopfield\_Fixed\_Act \hspace{0.5cm} & \hspace{0.5cm} for Hopfield networks\\
Hopfield\_Synchronous \hspace{0.5cm} & \hspace{0.5cm} for Hopfield networks\\
JE\_Order \hspace{0.5cm} & \hspace{0.5cm} for Jordan or Elman network\\
JE\_Special \hspace{0.5cm} & \hspace{0.5cm} for Jordan or Elman network\\
Kohonen\_Oder \hspace{0.5cm} & \hspace{0.5cm} for Self-Organizing Maps (SOMS)\\
RCC\_Order \hspace{0.5cm} & \hspace{0.5cm} for
reccurrent-Cascade-Correlation networks\\ 
Random\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Random\_Permutation \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Serial\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
Synchronous\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
TimeDelay\_Order \hspace{0.5cm} & \hspace{0.5cm} for Time-Delay networks\\
Topological\_Order \hspace{0.5cm} & \hspace{0.5cm} for any network\\
\hline
\end {tabular}
\end{center}

All these functions receive their input from the five update parameter fields
in the control panel. See figure \ref{ControlPanel} 

The following parameters are required for ART, Hopfield, and
Autoassociative networks. The description of the update functions will
indicate which parameter is needed.

\begin{tabbing}
XXXXXXX \= X \= X \= \kill
\> \\
\> $\rho$ \> = vigilance parameter with ($0 \leq \rho \leq 1.$) : field1\\
\> $\rho^a$ \> = initial vigilance parameter for the ART$^a$ part of the net\\ 
\> \> \> with (0 $< \rho^a <$ 1) : field1\\
\> $\rho^b$ \> = vigilance parameter for ART$^b$ with ($0 < \rho^b < 1$) : field2\\
\>  $\rho^c$ \> = Inter-ART-Reset control with ($0 < \rho^b < 1$) : field3\\
\> $a$ \> = strength of the influence of the lower level in F1 by the middle\\
\> \> \> level with ($a$ $>$ 0) : field2\\
\> $b$ \> = strength of the influence of the middle level in F1 by the upper\\
\> \> \> level with ($b$ $>$ 0) : field3\\
\> $c$ \> = part of the length of vector p with (0 $<$ c $<$ 1) : field4\\
\> $\Theta$ \> = Kind of threshold with ($0 \leq \Theta < 1.$) : field5\\
\> c \> = number of units : field1\\
\> n \> = iteration parameter : field1\\
\> x \> = number of selected neurons\\
\> \\
\end {tabbing}

Field1..Field5 are the positions in the control panel. For a more
detailed description of ART-parameters see section \ref{art models in
  snns}: ART Models in SNNS

Now here is a description of the steps the various update functions perform, 
and of the way in which they differ.



\subsubsection* {ART1\_Stable}
\index{ART1\_Stable}
The {\tt ART1\_Stable} update function updates the neurons activation
and output values until a stable state is reached. In one propagation
step the activation of all non-input units is calculated and then the
calculation of the output of all neuron follows. The state is
considered stable if the 'classifiable' or the 'not classifiable'
neuron is selected. 'Classifiable' means that the input vector
(pattern) is recognized by the net. 'Not classifiable' means that that
there is no neuron in the recognition layer which would fit the input
pattern. The required parameter is $\rho$ in field1.





\subsubsection* {ART1\_Synchronous}
\index{ART1\_Synchronous}
The algorithm of the {\tt ART1\_Synchronous} update function is the
ART1 equivalent to the algorithm of the Synchronous\_Order function.
The only difference is that the winner of the ART1 recognition layer
is identified. The required parameter is $\rho$ in field1.



\subsubsection* {ART2\_Stable}
\index{ART2\_Stable}
The first task of this algorithm is to initialize the activation of
all units. This is necessary each time a new pattern is loaded to the
network. The ART2 net is initialized for a new pattern now. The output
and activation will be updated with synchronous propagations until a
stable state is reached. One synchronous propagation cycle means that
each neuron calculate its output and then its new activation. The
required parameters are $\rho$, $a$, $b$, $c$, $\Theta$ in field1,
field2, field3, field4, field5 respectively.
 



\subsubsection* {ART2\_Synchronous}
\index{ART2\_Synchronous}
This function is the ART2 equivalent to the {\tt Synchronous\_Order}
function. The only difference is that additionally the winner neuron
of the ART1 recognition layer is calculated. The required parameters
are $\rho$, a, b, c, $\Theta$ in field1, field2, field3, field4,
field5 respectively.







\subsubsection* {ARTMAP\_Stable}
\index{ARTMAP\_Stable}
{\tt ARTMAP\_Stable} updates all units until a stable state is
reached. The state is considered stable if the classified or
unclassified unit is 'on'. All neurons compute their output and
activation in one propagation step. The propagation step continues
until the stable state is reached. The required parameters are
$\rho^a, \rho^b, \rho^c$ in field1, field2, field3 respectively.



\subsubsection* {ARTMAP\_Synchronous}
\index{ARTMAP\_Synchronous}
The first step is to calculate the output value of the input units 
(input units of ARTa, ARTb). Now a complete propagation step takes place, 
i.e. all units calculate their output and activation value. The 
search for two recognition neuron with highest activation follows. The search
takes place in both ARTa and ARTb. The required parameters are 
$\rho^a, \rho^b, \rho^c$ in field1, field2, field3 respectively.


\subsubsection* {Auto\_Synchronous}
\index{Auto\_Synchronous}
First the {\tt Auto\_Synchronous} function calculates the activation
of all neurons. The next step is to calculate the output of all units.
The two steps will be repeated n times. For the iteration parameter n,
which has to be provided in field1, a value of 50 has shown to be very
suitable.

\subsubsection* {BAM\_Order}
\index{BAM\_Order}
The first step of this update function is to search for the first
hidden unit of the network. The current output is saved and a new
output is calculated for all neurons of the hidden and output layer.
Once this is accomplished the next progression of the hidden and
output units starts. Now for each neuron of the hidden and output
layer the new output is saved and the old saved output is restored.
With this older output the activation of all hidden and output neurons
is calculated. After this task is accomplished the new saved output
value of all hidden and output neurons is restored.


\subsubsection* {BBTT\_Order}
\index{BBTT\_Order}
The {\tt BBTT\_Order} algorithm performs an update on a recurrent
network. The recurrent net can be transformed into a regular
feedforward net with an input, multiple hidden and output layer. At
the beginning the update procedure checks if there is a zero-input
pattern in the input layer. Suppose there is such a pattern, then the
so called {\tt i\_act} value buffer is set to 0 for all neurons.  In
this case {\tt i\_act} can be seen as a buffer for the output value of
the hidden and output neurons. The next step is to copy the {\tt
  i\_act} value to the output of all hidden and output neurons. The
new activation of the hidden and output units will be calculated. Now
the new output for every neuron in the hidden and output layer will be
computed and stored in {\tt i\_act}.






\subsubsection* {CC\_Order}
\index{CC\_Order}
The {\tt CC\_Order} update function propagates a pattern through the net. 
This means all neurons calculate their new activation and output in a 
topological order. The {\tt CC\_Order} update function also handles the 
special units which represent the candidate units.  

\subsubsection* {CounterPropagation}
\index{CounterPropagation}
The {\tt CounterPropagation} update algorithm updates a net that
consists of a input, hidden and output layer. In this case the hidden
layer is called the Kohonen layer and the output layer is called the
Grossberg layer.  At the beginning of the algorithm the output of the
input neurons is equal to the input vector. The input vector is
normalized to the length of one.  Now the progression of the Kohonen
layer starts. This means that a neuron with the highest net input is
identified. The activation of this winner neuron is set to 1. The
activation of all other neurons in this layer is set to 0.  Now the
output of all output neurons is calculated. There is only one neuron
of the hidden layer with the activation and the output set to 1. This
and the fact that the activation and the output of all output neurons
is the weighted sum on the output of the hidden neurons implies that
the output of the output neurons is the weight of the link between the
winner neuron and the output neurons. This update function makes sense
only in combination with the CPN learning function.
 

\subsubsection* {Dynamic\_LVQ}
\index{Dynamic\_LVQ}
This update algorithm initializes the output and activation value of
all input neurons with the input vector. Now the progression of the
hidden neurons begins.  First the activation and output of each of the
hidden neurons is initialized with 0 and the new activation will be
calculated. The hidden neuron with the highest activation will be
identified. Note that the activation of this winner unit has to be $>$
-1. The class which the input pattern belongs to will be propagated to
the output neuron and stored as the neurons activation.  This update
function is sensible only in combination with the DLVQ learning
function.

 
\subsubsection* {Hopfield\_Fixed\_Act}
\index{Hopfield\_Fixed\_Act}
This update function selects x neurons with the highest net-inputs and 
associates the activation value of those units with 1. The activation value
of all other units is associated with 0. Afterwards the output value of all
neurons will be calculated. The required parameter is x in field1.


\subsubsection* {Hopfield\_Synchronous}
\index{Hopfield\_Synchronous}
This update function calculates the output of all neurons first. This has 
to be done in order to propagate the pattern which is represented by the 
input vector. The activation update of all neurons which are not input 
neurons follows. The next step is to calculate the output value of those 
units. The input units are handled next. The activation of the input 
neurons is calculated and the next progression updates the output of all 
input units. 

 
\subsubsection* {JE\_Order}
\index{JE\_Order}
This update function propagates a pattern from the input layer to the
first hidden layer, then to the second hidden layer, etc. and finally
to the output layer. After this follows a synchronous update of all
context units. This function is makes sense only for JE-networks.



\subsubsection* {JE\_Special}
\index{JE\_Special}
Using the update function {\tt JE\_Special}, input patterns will
be generated dynamically. Let $n$ be the number of input units and $m$
the number of output units of the network.  {\tt JE\_Special} 
generates the new input vector with the output of the last $n-m$ input
units and the outputs of the $m$ output units. The usage of this
update function requires $n > m$. The propagation of the newly generated
pattern is done like using {\tt JE\_Update}.  The number of the actual
pattern in the control panel has no meaning for the input pattern when
using {\tt JE\_Special}. This update function is used to determine the 
prediction capabilities of a trained network.


\subsubsection* {Kohonen\_Order}
\index{Kohonen\_Order}
The {\tt Kohonen\_Order} function propagates neurons in a topological order.
 There are 2 propagation steps. The first step all input units are 
propagated, which means that the output of all neurons is calculated. The 
second step consists of the propagation of all hidden units. This 
propagation step calculates all hidden neuron's activation and output. Please
note that the activation and output are normally not required for the
Kohonen algorithm. The activation and output values are used for display and 
evaluation reasons internally. The {\tt Act\_Euclid} activation function for 
example, copies the Euclidian distance of the unit from the training pattern 
to the units activation.

\subsubsection* {RCC\_Order}
\index{RCC\_Order}
This function is similar to the {\tt CC\_Order} except it takes special care 
of the recurrent links. Use this function only for RCC-networks.


\subsubsection* {Random\_Order}
\index{Random\_Order}
The {\tt Random\_Order} update function selects a neuron and
calculates its activation and output value. The selection process is
absolutely random and will be repeated n times. The parameter n is the
number of existing neurons. One specific neuron can be selected more
than one time while other neurons may be left out. This kind of update
function is rarely used and is just a theoretical base to prove the
stability of Hopfield nets.



\subsubsection* {Random\_Permutation}
\index{Random\_Permutation}
This update function is similar to the {\tt Random\_Order} function. The only
difference is that a random permutation of all neurons is used to select 
the order of the units. This guarantees that each neuron will be selected 
exactly once to calculate the output and activation value. This procedure 
has two big disadvantages. The first disadvantage is that the computation
of the permutation is very time consuming and the second disadvantage is 
that it takes a long time until a stable output vector has been established. 

\subsubsection* {Serial\_Order}
\index{Serial\_Order}
The {\tt Serial\_Order} update function calculates the activation and output 
value for each unit. The progression of the neurons is serial which means the 
computation process starts at the first unit and proceeds to the last one.



\subsubsection* {Synchronous\_Order}
\index{Synchronous\_Order}
With the synchronous update function all neurons change their value at the same
time. All neurons calculate their activation in one single step. The output 
of all neurons will be calculated after the activation step. The difference to 
the {\tt serial\_order} update function  is that the calculation of the 
output and activation value requires two progressions of all neurons.
This kind of propagation is very useful for distributed systems (SIMD). 

 
\subsubsection* {TimeDelay\_Order}
\index{TimeDelay\_Order}
The update function TimeDelay\_Order is used to propagate patters
through a time delay network. Its behavior is analogous to the
Topological\_Order functions with recognition of logical links.


  
\subsubsection*  {Topological\_Order}
\index{Topological\_Order}
This mode is the most favorable mode for feedforward nets. The neurons
calculate their new activation in a topological order. The topological order is
given by the net-topology. This means that the first processed layer 
is the input layer. The next processed layer is the first hidden layer 
and the last layer is the output layer. A learning cycle is defined as a pass
through all neurons of the net. Shortcut-connections are allowed.




\section {Initialization Functions}
\label{InitFuncs}
\index{Initialization Functions}

In order to work with various neural network models and learning
algorithms, different initialization functions that initialize the
components of a net are required. Backpropagation, for example, will
not work properly if all weights are initialized to the same value. 

To select an initialization function, one must click \fbox {\tt
SEL. FUNC} in the {\tt INIT} line of the control panel.

The following initialization functions are available:

\begin{center}
\begin{tabular}{|l|l|}
\hline
ART1\_Weights \hspace{0.5cm}         & for ART1 networks\\
ART2\_Weights \hspace{0.5cm}         & for ART2 networks\\
ARTMAP\_Weights \hspace{0.5cm}       & for ARTMAP networks\\
CC\_Weights \hspace{0.5cm}           & for Cascade Correlation networks\\
ClippHebb \hspace{0.5cm}             & for Associative Memory networks\\
CPN\_Rand\_Pat \hspace{0.5cm}        & for Counterpropagation\\
CPN\_Weights\_v3.2 \hspace{0.5cm}    & for Counterpropagation\\
CPN\_Weights\_v3.3 \hspace{0.5cm}    & for Counterpropagation\\
DLVQ\_Weights \hspace{0.5cm}        & for Dynamic Learning Vector Quantization\\
Hebb \hspace{0.5cm}                  & for Associative Memory networks\\
Hebb\_Fixed\_Act \hspace{0.5cm}      & for Associative Memory networks\\
JE\_Weights \hspace{0.5cm}           & for Jordan or Elman networks\\
Kohonen\_Rand\_Pat \hspace{0.5cm}    & for Self-Organizing Maps (SOMS)\\
Kohonen\_Const \hspace{0.5cm}        & for Self-Organizing Maps (SOMS)\\
Kohonen\_Weights\_v3.2 \hspace{0.5cm}& for Self-Organizing Maps (SOMS)\\
Pseudoinv \hspace{0.5cm}             & for Associative Memory networks\\
Randomize\_Weights \hspace{0.5cm}    & for any network, except the ART-family\\
Random\_Weights\_Perc \hspace{0.5cm} & for Backpercolation\\ 
RBF\_Weights \hspace{0.5cm}          & for Radial Basis Functions (RBFs)\\
RBF\_Weights\_Kohonen \hspace{0.5cm} & for Radial Basis Functions (RBFs)\\
RBF\_Weights\_Redo \hspace{0.5cm}    & for Radial Basis Functions (RBFs)\\
RCC\_Weights \hspace{0.5cm}      & for recurrent Cascade-Correlation networks\\
RM\_Random\_Weights \hspace{0.5cm}   & for Autoassociative Memory Networks\\
\hline
\end {tabular}
\end {center}



All these functions receive their input from the five init parameter fields
in the control panel. See figure \ref {ControlPanel} 

Here is a short description of the different initialization functions:


\subsubsection*{ART1\_Weights}
\index{ART1\_Weights}
ART1\_Weights is responsible to set the initial values of the trainable
links in an ART1 network. These links are the ones
from {F$_1$} to {F$_2$} and the ones from {F$_2$} to {F$_1$} respectively.
For more details see chapter \ref{ART1}.


\subsubsection* {ART2\_Weights}
\index{ART2\_Weights}
For an ART2 network the weights of the top-down-links
({F$_2$} $\rightarrow$ {F$_1$} links) are set to 0.0 according to the
theory (\cite{CG87b}). The choice of the initial bottom-up-weights is 
described in chapter \ref{ART2}.
 

\subsubsection* {ARTMAP\_Weights}
\index{ARTMAP\_Weights}
The trainable weights of an ARTMAP network are primarily the
ones of the two ART1 networks {ART$^a$} and {ART$^b$}, therefore the 
initialization process is similar. For more details see chapter \ref{ART1} 
and chapter \ref{ART2}.


\subsubsection* {CC\_Weights}
\index{CC\_Weights}
{\tt CC\_Weights} calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.


\subsubsection*{ClippHebb}
\index{ClippHebb}
The {\tt ClippHebb} algorithm is almost the same as the {\tt Hebb}
algorithm, the only difference is that all weights can only be set 
to 1 and 0. After the activation for the neurons is calculated, all 
weights $>$ 1 will be set to 1. 
As mentioned in \ref{Hebb} the {\tt ClippHebb} algorithm is a learning 
algorithm.


\subsubsection*{CPN\_Rand\_Pat}
\index{CPN\_Rand\_Pat}
This Counterpropagation initialization function initializes all weight
vectors of the Kohonen layer with random input patterns from the
training set. This guarantees that the Kohonen layer has no dead
neurons.

The weights of the Grossberg layer are all initialized to 1.


\subsubsection*{CPN\_Weights\_v3.2}
\index{CPN\_Weights\_v3.2}
This function generates random points in an n-dimensional hypercube
and later projects them onto the surface of an n-dimensional unit
hypersphere or onto one of its main diagonal sectors (main diagonal
quadrant for $n=2$, octant for $n=3$, ...).

First the interval, from which the Kohonen weights for the
initialization tasks are selected, is determined. Depending upon the
initialization parameters, which have to be provided in field1
and field2, the interval may be $[-1\,;+1]$, $[0\,;+1]$, or
$[-1\,;0]$.

Every component $w_{ij}$ of every Kohonen layer neuron j is then
assigned a random value from the above interval, yielding weight
vectors $\vec {w_{j}}$, which are random points within an
n-dimensional hypercube.  

The length of each vector $\vec {w_{j}}$ is then normalized to 1.

All weights of the neurons of the Grossberg layer are set to 1.  

Note that this initialization function does NOT produce weight vectors
with equal point density on the hypersphere, because with increasing
dimension of the hypercube in which the random dots are generated,
many more points are originally in the corners of the hypercube than
in the interior of the inscribed hypersphere.


\subsubsection*{CPN\_Weights\_v3.3}
\index{CPN\_Weights\_v3.3} This function generates random points in an
n-dimensional cube, throws out all vectors with length $>1$ and
projects the remaining onto the surface of an n-dimensional unit
hypersphere or onto one of its main diagonal sectors (main diagonal
quadrant for $n=2$, octant for $n=3$, ...).

First the interval, from which the Kohonen weights for the
initialization tasks are selected, is determined. Depending upon the
initialization parameters, which have to be provided in field1
and field2, the interval may be $[-1\,;+1]$, $[0\,;+1]$, or
$[-1\,;0]$.

Every component $w_{ij}$ of every Kohonen layer neuron j is then
assigned a random value from the above interval, yielding weight
vectors $\vec {w_{j}}$, which are random points within an
n-dimensional hypercube. If the weight vector $\vec {w_{j}}$ thus
generated is outside the unit hypersphere or hypersphere sector, a new
random vector is generated until eventually one is inside the
hypersphere or hypersphere sector.
Finally, the length of each vector $\vec {w_{j}}$ is normalized to 1.

The Grossberg layer weight vector components are all set to 1.

Note that this initialization function DOES produce weight vectors
with equal point density on the hypersphere. However, the fraction of
points from the hypercube which are inside the enscribed hypersphere
decreases exponentially with increasing vector dimension, thus
exponentially increasing the time to perform the initialization. This
method is thus only suitable for input dimensions up to 12 - 15.
(read Hecht-Nielsen: Neurocomputing, chapter 2.4, pp. 41 ff. for an 
interesting discussion on n-dim. geometry).


\subsubsection* {DLVQ\_Weights}
\index{DLVQ\_Weights}
{\tt DLVQ\_Weights} calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.


\subsubsection*{Hebb}
\index{Hebb}
\label{Hebb}
This procedure is similar to the Hebbian Learning Rule with a learning 
rate of 1. Additionally the bias of all input and output neurons is 
set with the parameters p1 and p2 which have to be provided in field1 
and field2. Please note that the {\tt Hebb}, {\tt ClippHebb}, {\tt HopFixAct}
and {\tt PseudoInv} initialization functions are actually learning functions.
The reason why those functions are called initialization functions, is the 
fact that there is no true training because all weights will be calculated 
directly. In case the values of the parameters p1 and p2 are 1 and -1
the bias of the input and output neurons will be set to ld (n) and ld (k).
Where n is the number of input neurons and k is the number of output neurons.
These settings are also the default settings for p1 and p2. 
In any other case the p1 and p2 represent the bias of the input and 
output neurons without any modification.


\subsubsection*{Hebb\_FixAct}
\index{Hebb\_FixAct}
This rule is necessary in order to do 'one-step-recall' simulations.
For more informations see \cite {Amari89}. 
For the calculation the bias following facts are assumed:

\begin {itemize}
\item {The implemented net is an autoassociative net}\\ 
      The neurons of an autoassociative net have to be input and output 
      neurons at the same time. A Hopfield network would be an example for 
      such a net.

\item {Fixed number of 1s}\\
      The patterns which are to be saved have a fixed 
      number of 1s.
\end {itemize}

The parameter h1 and h2 are required. Where h1 is the number of ones per 
pattern and h2 is the probable degree of distortion in percent. The parameters 
have to be inserted in field1 and field2.
This initialization function should be used only in connection with the 
{\tt Hopfield\_Fixed\_Act} update function. As mentioned in section \ref{Hebb} 
the {\tt Hebb\_FixAct} algorithm is a learning algorithm.



\subsubsection* {JE\_Weights}
\index{JE\_Weights}
This network consists of two types of neurons. The regular neurons and 
the so called context neurons. In such networks all links leading to 
context units are considered recurrent links.
The initialization function {\tt JE\_Weights} requires the
specification of five parameters:
 

\begin{itemize}
 \item \makebox[0.7cm][l]{$\alpha$, $\beta$} : 
       The weights of the forward connections are randomly chosen
       from the interval $[\alpha\,;\beta]$. $\alpha, \beta$ have to be 
       provided in field1 and field2 of the init panel.  
 \item \makebox[0.7cm][l]{$\lambda$} : 
       Weights of self recurrent links from context units to themselves.
       Simple Elman networks use $\lambda = 0$. {$\lambda$} has to be 
       provided in field3 of the init panel. 
 \item \makebox[0.7cm][l]{$\gamma$} :
       Weights of other recurrent links to context units. This value
       is often set to $1.0$. {$\gamma$} has to be provided in field4 
       of the init panel.
 \item \makebox[0.7cm][l]{$\psi$}  : 
       Initial activation of all context units. {$\psi$} has to be provided
       in field5 of the init panel.
\end{itemize}

Note that it is required that {$\alpha > \beta$}. If this is not the
case, an error message will appear on the screen.  The context units
will be initialized as described above. For all other neurons the bias
and all weights will be randomly chosen from the interval
$[\alpha\,;\beta]$.


\subsubsection*{Kohonen\_Weights\_v3.2} 
\index{Kohonen\_Weights\_v3.2} 
This initialization function is identical to CPN\_Weights\_v3.2 except
that it only initializes the Kohonen layer, because there is no second
Grossberg layer as in Counterpropagation.


\subsubsection*{Kohonen\_Const}
\index{Kohonen\_Const} 
Each component $w_{ij}$ of each Kohonen weight vector $w_{j}$ is set
to the value of $\frac{1}{\sqrt{n}}$, thus yielding all identical
weight vectors $w_{j}$ of length 1. This is no problem, because the
Kohonen algorithm will quickly pull weight vectors away from this
central dot and move them into the proper direction.


\subsubsection*{Kohonen\_Rand\_Pat}
\index{Kohonen\_Rand\_Pat}

This initialization function initializes all weight vectors of the
single Kohonen layer with random input patterns from the training
set. This guarantees that the Kohonen layer initially has no dead
neurons.


\subsubsection*{PseudoInv}
\index{PseudoInv}
The {\tt PseudoInv} initialization function computes all weights with 
the help of the pseudo inverse weight matrix  which is calculated with 
the algorithm of Greville. The formula for the weight calculation 
is: W = Q{$S^+$}. Where {$S^+$} is the 'Pseudoinverse' of the input 
vectors, Q are the output vectors and W are the desired weights of the net.
 The bias is not set and there are no parameters necessary. 
Please note that the calculated weights are usually odd. 
As mentioned in \ref{Hebb} the {\tt PseudoInv} algorithm is a learning 
algorithm.


\subsubsection* {Randomize\_Weights}
\index{Randomize\_Weights}
This function initializes all weights and the bias with distributed 
random values. The values are chosen from the interval $[\alpha\,;\beta]$. 
$\alpha$ and $\beta$ have to be provided in field1 and field2 of the 
init panel. It is required that $\alpha > \beta$.
 
\subsubsection* {Random\_Weights\_Perc}
\index{Random\_Weights\_Perc}
The first task of this function is to calculate the number of 
incoming links of a unit.
Once this is accomplished, the range of possible weight-values 
will be determined. The range will be calculated with the $\alpha$
and $\beta$ parameters, which have to be provided with the help of 
the init panel in field1 and field2. 
If $\alpha = \beta$ all weights and the bias will be set to 
the value of the $\alpha$ parameter. If $\alpha <> \beta$ the links of 
all neurons will be initialized with random values selected from the interval
$[\alpha\,;\beta]$ and divided by the number of incoming links of every 
neuron. The bias will also be set to zero.


\subsubsection* {RBF\_Weights}
\index{RBF\_Weights}
This procedure first selects evenly distributed centers $\vec{t_j}$
from the loaded training patterns and assigns them to the links between 
input and hidden layer. Subsequently the bias of all neurons (parameter $p$)
inside the hidden layer is set to a value determined by the user and finally 
the links between hidden and output layer are computed. For more details
see chapter \ref{RBF}.



\subsubsection* {RBF\_Weights\_Kohonen}
\index{RBF\_Weights\_Kohonen}
Using the self-organizing method of Kohonen feature maps, appropriate 
centers are generated on base of the teaching patterns. The computed 
centers are copied into the corresponding links. No other links and bias 
are changed. For more details see chapter \ref{RBF}.



\subsubsection* {RBF\_Weights\_Redo} 
\index{RBF\_Weights\_Redo} 
In contrast to the preceding procedure only the links between hidden and 
output layer are computed. All other links and bias remain unchanged.
For more details see chapter \ref{RBF}.

\subsubsection* {RCC\_Weights}
\index{RCC\_Weights}
RCC\_Weights calls the {\tt Randomize\_Weights} function. 
See {\tt Randomize\_Weights}.


\subsubsection* {RM\_Random\_Weights}
\index{RM\_Random\_Weights}
The {\tt RM\_Random\_Weights} function initializes the bias and all
weights of all units which are not input-units with a random value.
This value is selected from the interval $[\alpha\,;\beta]$. $\alpha$
and $\beta$ have to be provided in field1 and field2 of the init
panel. $\alpha > \beta$ has to hold.



\section{Creating and Editing Unit Prototypes and Sites}
\label{EditFTypes}
\index{F-Type}
\index{Prototypes}
\index{Site}

\begin{figure}[htbp]
  \begin{center}
    \begin{minipage}[htbp]{8cm}
      \screendump{0.6}{xgui_figs/FigEditPanelFTypes.ps}
    \end{minipage}
    \hfill
    \begin{minipage}[htbp]{6cm}
      \screendump{0.6}{xgui_figs/FigEditPanelSites.ps}
    \end{minipage}
    \caption{Edit panels for unit prototypes (f-types) and sites}
    \label{FigEditPanelFTypes}
  \end{center}
\end{figure}

\index{Edit F-Types Panel}
\index{Panel!Edit F-Types}
\index{Edit Sites Panel}
\index{Panel!Edit Sites}
Figure~\ref{FigEditPanelFTypes} shows the panels to edit unit
prototypes (f-types) and sites. The change of the f-type is performed
on all units of that type. Therefore, the functionality of all units
assigned to an f-type can easily be changed. The elements in the panel
have the following meaning:


\begin{itemize}

\item \fbox{\tt SELECT}: Selects of the activation and output function.

\item \fbox{\tt CHOOSE}: Chooses the f-type to be changed.

\item \fbox{\tt SET}: Makes the settings/changes permanent. Changes in
the site list are not set (see below).

\item \fbox{\tt NEW}, \fbox{\tt DELETE}: Creates or deletes an f-type.

\item \fbox{\tt ADD}, \fbox{\tt DELETE}: F-types also specify the
sites of a unit. Therefore these two buttons are necessary to
add/delete a site in the site list.

\end{itemize}

Note: The number and the selection of sites can not be changed 
after the creation of an f-type.

The elements in the edit panel for sites are almost identical. A site
is selected for change by clicking at it in the site list.

\begin{itemize}

\item \fbox{\tt SELECT}: Selects the new site function. The change is
performed in all sites in the net with the same name.

\item \fbox{\tt SET}: Validates changes/settings.

\item \fbox{\tt NEW}: Creates a new site.

\item \fbox{\tt DELETE}: Deletes the site marked in the site list.

\end{itemize}

