\chapter {Network Analyzing Tools}
\label{xguiAnalyze}
\thispagestyle{empty}

\section{Inversion}

Very often the user of a neural network asks what properties an input
pattern must have in order to let the net generate a specific
output. To help answer this question, the Inversion algorithm
developed by J.~Kindermann and A.~Linden (\cite{KinLin90}) was
implemented in SNNS.  

\subsection{The Algorithm}
\index{Inversion!Algorithm}

The inversion of a neural net tries to find an input pattern that
generates a specific output pattern with the existing connections. To
find this input, the deviation of each output from the desired output
is computed as error $\delta$. This error value is used to approach
the target input in input space step by step. Direction and length of
this movement is computed by the inversion algorithm.

\index{Least Mean Square Error}
The most commonly used error value is the {\sl Least Mean Square
Error}. $E^{LMS}$ is defined as

\begin{displaymath}
E^{LMS} = \sum_{p=1}^{n} [T_p - f(\sum_i w_{ij} o_{pi})]^2
\end{displaymath}

The goal of the algorithm therefore has to be to minimize $E^{LMS}$.

Since the error signal $\delta_{pi}$ can be computed as 

\begin{displaymath}
\delta_{pi} = o_{pi} (1 - o_{pi}) \sum_{k \in Succ(i)} \delta_{pk} w_{ik}
\end{displaymath}

and for the adaption value of the unit activation follows

\begin{displaymath}
\bigtriangleup net_{pi} = \eta \delta_{pi} \quad \mbox{resp.}\quad net_{pi} =
net_{pi} + \eta \delta_{pi}
\end{displaymath}


In this implementation, a uniform pattern is applied to the input
units in the first step, whose activation level depends upon the
variable {\tt input pattern}. This pattern is propagated through the
net and generates the initial output $O^{(0)}$. The difference between
this output vector and the target output vector is propagated
backwards through the net as error signals $\delta_{i^{(0)}}$. This is
analogous to the propagation of error signals in the backpropagation
training, with the difference that no weights are adjusted here. When
the error signals reach the input layer, they represent a gradient in
input space, which gives the direction for the gradient descent.
Thereby, the new input vector can be computed as

\begin{displaymath}
I^{(1)} = I^{(0)} + \eta * \delta_{i^{(0)}}
\end{displaymath}

where $\eta$ is the step size in input space, which is set by the
variable {\tt eta}.

This procedure is now repeated with the new input vector until the
distance between the generated output vector and the desired output
vector falls below the predefined limit of {\tt delta\_max}, when the
algorithm is halted.

For a more detailed description of the algorithm and its
implementation see \cite{Mamier92}.


\subsection{Inversion Display}
\index{Inversion!Display}
\index{Display!Inversion Display}

The inversion algorithm is called by clicking the \fbox{\tt INVERSION}
button in the manager panel.

Picture~\ref{invdispl1} shows an example of the generated display.

\begin{figure}[htb]
\begin{center}

  \begin{minipage}[htb]{\textwidth}
    \screendump{0.6}{xgui_figs/invdispl_raw.ps}
  \end{minipage}
  \caption{\label{invdispl1}The Inversion Display}

\end{center}
\end{figure}

The display consists of two regions. The larger, lower part contains a
sketch of the input and output units of the network, while the upper
line holds a series of buttons. Their respective functions are:

%\begin{list}{BLANK}{\labelwidth1.6cm \leftmargin2cm \labelsep0.4cm
%\itemsep0.5cm \parsep0.3cm \listparindent0cm \topsep0.5cm}

\begin{enumerate}

\item \fbox{\tt DONE}: Quits the inversion algorithm and closes the
display. 

\item \fbox{\tt STEP}: Starts / Continues the algorithm. The program
starts iterating by slowly changing the input pattern until either the
STOP button is pressed, or the generated output pattern approximates
the desired output pattern sufficiently well. Sufficiently well means
that all output units have an activation which differs from the
expected activation of that unit by at most a value of $\delta_{max}$.
This error limit can be set in the setup panel (see below). During the
iteration run, the program prints status reports to stdout.
\index{Inversion!Error}

{\tt
\begin{verbatim}
    cycle 50 inversion error 0.499689 still 1 error unit(s)
    cycle 100 inversion error 0.499682 still 1 error unit(s)
    cycle 150 inversion error 0.499663 still 1 error unit(s)
    cycle 200 inversion error 0.499592 still 1 error unit(s)
    cycle 250 inversion error 0.499044 still 1 error unit(s)
    cycle 269 inversion error 0.000000 0 error units left
\end{verbatim}
}

where cycle is the number of the current iteration, inversion error is
the sum of the squared error of the output units for the current input
pattern, and error units are all units that have an activation that
differs more than the value of  $\delta_{max}$ from the target activation.

\item \fbox{\tt STOP}: Interrupts the iteration. The status of the network
remains unchanged. The interrupt causes the current activations of the
units to be displayed on the screen. A click to the \fbox{STEP} button
continues the algorithm from its last state.  Alternatively the
algorithm can be reset before the restart by a click to the \fbox{NEW}
button, or continued with other parameters after a change in the
setup. Since there is no automatic recognition of infinite loops in
the implementation, the \fbox{STOP} button is also necessary when the
algorithm obviously does not converge.

\item \fbox{\tt NEW} Resets the network to a defined initial status.  All
variables are assigned the values in the setup panel.  The iteration
counter is set to zero.

%\clearpage
\item \fbox{\tt SETUP}: Opens a pop-up window to set all variables
associated with the inversion. These variables are:
\index{Inversion!Variable}

\begin{tabular}{ll}

{\tt eta} & The step size for changing the activations. It should
range\\ & from 1.0 to 10.0. Corresponds to the learning
factor in\\ & backpropagation.\\

{\tt delta\_max} & The maximum activation deviation of an
output unit.\\ & Units with higher deviation are called error units.\\
& A typical value of delta\_max is 0.1.\\

{\tt Input pattern}
                 & Initial activation of all input units.\\
{\tt 2nd approx ratio}
                 & Influence of the second approximation. Good values
range\\ & from 0.2 to 0.8.\\
\end{tabular}

A short description of all these variables can be found in an
associated help window, which pops up on pressing \fbox{HELP} in the
setup window.

The variable {\tt second approximation} can be understood as follows:
Since the goal is to get a desired output, the first approximation is
to get the network output as close as possible to the target output.
There may be several input patterns generating the same output. To
reduce the number of possible input patterns, the second approximation
specifies a pattern the computed input pattern should approximate as
well as possible. For a setting of 1.0 for the variable {\tt Input
pattern} the algorithm tries to keep as many input units as possible
on a high activation, while a value of 0.0 increases the number
of inactive input units. The variable {\tt 2nd approx ratio} defines
then the importance of this input approximation.

It should be mentioned, however, that the algorithm is very unstable.
One inversion run may converge, while another with only slightly
changed variable settings may run indefinitely. The user therefore may
have to try several combinations of variable values before a
satisfying result is achieved. In general, the better the net was
previously trained, the more likely is a positive inversion result.


\item \fbox{\tt HELP}: Opens a window with a short help on handling the
inversion display.

\end{enumerate}

The network is displayed in the lower part of the window according to
the settings of the last opened 2D--display window. Size, color, and
orientation of the units are read from that display pointer. 

\subsection{Example Session}
\index{Example!Inversion} 
\index{Inversion!Example}

The inversion display may be called before or after the network has
been trained. A pattern file for the network has to be
loaded prior to calling the inversion. A target output of the network
is defined by selecting one or more units in the 2D--display by
clicking the middle mouse button. After setting the variables in the
setup window, the inversion run is started by clicking the start
button.  At regular intervals, the inversion gives a status report on
the shell window, where the progress of the algorithm can be observed.
When there are no more error units, the program terminates and the
calculated input pattern is displayed. If the algorithm does not
converge, the run can be interrupted with the stop button and the
variables may be changed. The calculated pattern can be tested for
correctness by selecting all input units in the 2D--display and then
deselecting them immediately again. This copies the activation of the
units to the display. It can then be defined and tested with the usual
buttons in the control panel. The user is advised to delete the
generated pattern, since its use in subsequent learning cycles alters
the behavior of the network which is generally not desirable.

Figure~\ref{invdispl} shows an example of a generated input pattern (left).
Here the minimum active units for recognition of the letter 'V' are
given. The corresponding original pattern is shown on the right.

\clearpage
\begin{figure}[htb]
\begin{center}

  \begin{minipage}[htb]{9cm}
    \screendump{0.5}{xgui_figs/invdispl1.ps}
  \end{minipage}
  \hfill
  \begin{minipage}[hbt]{5cm}
    \screendump{0.5}{xgui_figs/letter_v.ps}
  \end{minipage}
  \caption{\label{invdispl}An Example of an Inversion Display (left)
    and the original pattern for the letter V}

\end{center}
\end{figure}


\section{Network Analyzer} 
\index{Network Analyzer}

The network analyzer is a tool to visualize different types 
of graphs. An overview of these graphs is shown in table \ref{na_tab}. 
This tool was especially developed for the prediction of time series with 
partial recurrent networks, but is also useful for regular feedforward
networks. Its window is opened by selecting the entry {\tt ANALYZER}
in the menu under the  \fbox{\tt GUI} button.

The x-y graph is used to draw the activations or outputs of two units 
against each other. The t-y graph displays the activation (or output) 
of a unit during subsequent discrete time steps. The t-e graph makes it 
possible to visualize the error during subsequent discrete time steps.

\begin{table}[htb]
 \begin{center}
  \begin{tabular}{|l|l|} \hline
   Type  & Axes \\ \hline \hline
   x--y  & \makebox[1.7em][l]{hor.}: 
           activation or output of a unit $x$\\
         & \makebox[1.7em][l]{ver.}: 
           activation or output of a unit $y$\\ \hline
   t--y  & \makebox[1.7em][l]{hor.}: 
           time $t$\\
         & \makebox[1.7em][l]{ver.}: 
           activation or output of a unit $y$\\ \hline 
   t--e  & \makebox[1.7em][l]{hor.}: 
           time $t$\\
         & \makebox[1.7em][l]{ver.}: 
           error $e$\\ \hline 
  \end{tabular} 
  \caption{The different types of graphs, which can be visualized with the
           network analyzer.}
  \label{na_tab}
 \end{center}
\end{table}

%\vfill
\begin{figure}[thb]
 \begin{center}
  \begin{minipage}[htb]{\textwidth}
    \screendump{0.6}{nn_models/net_analyzer.ps}
  \end{minipage}
  \caption{The Network Analyzer Window.} 
    \label{net_analyzer}
 \end{center}
\end{figure}

On the right side of the window, there are different buttons with the 
following functions:

\framebox[2cm][c]{\tt ON}     \quad : \hfill
 \begin{minipage}[t]{4.7in}
  This button is used to ''switch on'' the network analyzer. If the 
  Network Analyzer is switched on, every time
  a pattern has been propagated through the network, the network analyzer
  updates its display.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt LINE}   \quad : \hfill
 \begin{minipage}[t]{4.7in}
  The points will be connected by a line, if this button is toggled.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt GRID}   \quad : \hfill 
 \begin{minipage}[t]{4.7in}
  Displays a grid.
  The number of rows and columns of the grid can be specified in the 
  network analyzer setup.   
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt CLEAR}  \quad : \hfill 
 \begin{minipage}[t]{4.7in}
  This button clears the graph in the display. The time counter will be
  reset to 1. If there is an active {\tt M--TEST} operation, this operation
  will be killed.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt M-TEST} \quad : \hfill
 \begin{minipage}[t]{4.7in}
  A click on this button corresponds to several clicks on the 
  \fbox{\tt TEST} button in the control panel. The number $n$ of {\tt TEST}
  operations to be executed can be specified in the Network Analyzer setup.
  Once pressed, the button remains active until all $n$ {\tt TEST} 
  operations have been executed or the {\tt M--TEST} operation has been 
  killed, e.g. by clicking the \fbox{STOP} button in the control panel.  
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt RECORD} \quad : \hfill 
 \begin{minipage}[t]{4.7in}
  If this button activated, the points will not only be shown on the
  display, but their coordinates will also be saved in a file. The name of 
  this file can be specified in the setup of the Network Analyzer.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt D-CTRL}  \quad : \hfill 
 \begin{minipage}[t]{4.7in}
  Opens the display control window of the Network Analyzer.
  The description of this window follows below.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt SETUP}  \quad : \hfill 
 \begin{minipage}[t]{4.7in}
  Opens the Network Analyzer setup window. The description of the setup 
  follows in the next subsection.
 \end{minipage}

\medskip

\framebox[2cm][c]{\tt DONE}   \quad : \hfill  
 \begin{minipage}[t]{4.7in}
  Closes the network analyzer window. An active {\tt M--TEST} operation
  will be killed.
 \end{minipage}

\bigskip



\subsection{The Network Analyzer Setup}

The setup window can be opened by pressing the {\tt SETUP} button on 
the right side of the network analyzer window.
The shape of the setup window depends on the type of graph to display
(see fig. \ref{na_setup}).

\begin{figure}[htbp]
 \begin{center}
 \begin{picture}(400,540)(0,0)
  \put(  0,360){\mbox{\screendump{0.6}{xgui_figs/na_setup1.ps}}}
  \put(240,190){\mbox{\screendump{0.6}{xgui_figs/na_setup2.ps}}}
  \put(  0,  0){\mbox{\screendump{0.6}{xgui_figs/na_setup3.ps}}}

  \put(220,515){\vector(-1, 0){60}} 
  \put(220,483){\vector(-4, 1){60}}
  \put(220,430){\vector(-4, 1){60}}
  \put(192,269){\vector( 4,-1){60}}
  \put(220,115){\vector(-4,-1){60}}
  \put(220, 57){\vector(-4,-1){60}}
 
  \put(235,515) {\circle{18}} \put(232,511){1}
  \put(235,483) {\circle{18}} \put(232,479){2}
  \put(235,430) {\circle{18}} \put(229,426){3a}
  \put(177,269) {\circle{18}} \put(174,265){4}
  \put(235,115) {\circle{18}} \put(229,111){3b}
  \put(235, 57) {\circle{18}} \put(232, 53){5}

 \end{picture}
 \caption[The Network Analyzer Setup Windows]
         {The Network Analyzer SetupWindows:
          the setup window for a x-y--graph (top), the setup window for a 
          t-y--graph (middle) and the setup window for a t-e--graph (bottom).}
 \label{na_setup}
 \end{center}
\end{figure}

The setup window consists of five parts (see fig. \ref{na_setup}):

\medskip

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(8,16){1}
 \end{picture}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 To select the type of graph (see table \ref{na_tab}) to be displayed, 
 press the corresponding button \fbox{\tt X-Y} , \fbox{\tt T-Y} or
 \fbox{\tt T-E}. 
\end{minipage}

\bigskip

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(8,16){2}
 \end{picture}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 The second part of the setup window is used to specify some attributes
 about the axes. The first line contains the values for the axes in 
 horizontal direction, the second line these for the vertical axes.
 The columns {\tt min} and {\tt max} define the area to be displayed.
 The numbers of the units, whose activation or output values should be 
 drawn have to be specified in the column {\tt unit}. The last column
 {\tt grid} the number of columns and rows of the grid can be varied. 
 The labeling of the axes is dependent on these values, too.   
\end{minipage}

\bigskip

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(5,16){3a}
 \end{picture}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 The selection between showing the activation or the output of a unit along the
 x-- or y--axes can be made here.
 To draw the output of a unit click on \fbox{\tt OUT} and to draw the
 activation of a unit click on \fbox{\tt ACT}.
\end{minipage}

\bigskip

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(5,16){3b}
 \end{picture}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 Different types of error curves can be drawn:

 \begin{minipage}[t]{2.5cm}
  \vphantom{x} \vspace{0.3ex}
  \framebox[2.5cm][c]{$\displaystyle \sum\limits_{i} |t_{i} - o_{i}|$}
 \end{minipage}
 \ \hfill \ 
 \begin{minipage}[t]{4.0in}  
  \vspace{0.3ex}
  For each output unit the difference between the generated output and
  the teaching output is computed. The error is computed as the sum of
  the absolute values of the differences. If \fbox{\tt AVE} is toggled,
  the result is divided by the number of output units, giving the
  average error per output unit.
 \end{minipage}
\end{minipage}

\medskip 

\begin{minipage}[t]{0.4in}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 \begin{minipage}[t]{2.5cm}
  \vphantom{x} \vspace{0.3ex}
  \framebox[2.5cm][c]{$\displaystyle \sum\limits_{i} |t_{i} - o_{i}|^{2}$}
 \end{minipage}
 \ \hfill \
 \begin{minipage}[t]{4.0in}
  \vspace{0.3ex}
  The error is computed as above, but the square of the differences is
  taken instead of the absolute values. With \fbox{\tt AVE} the mean
  squared deviation is computed.
 \end{minipage}
\end{minipage}

\medskip 

\begin{minipage}[t]{0.4in}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 \begin{minipage}[t]{2.5cm}
  \vphantom{x} \vspace{0.3ex}
  \framebox[2.5cm][c]{\vphantom{\Huge Q} 
                      $^{\displaystyle |t_{j} - o_{j}|}$ 
                      \vphantom{\Huge Q}}
 \end{minipage}
 \ \hfill \
 \begin{minipage}[t]{4.0in}
  \vspace{0.3ex}
  Here the deviation of only a single output unit is processed. The
  number of the unit is specified as  {\tt unit j}.
 \end{minipage}
\end{minipage}

\bigskip 

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(8,16){4}
 \end{picture}
\end{minipage}
\ \hfill \
\begin{minipage}[t]{5.4in}
 \begin{minipage}[t]{0.7in}
  {\tt m-test}: 
 \end{minipage}
 \ \hfill \
 \begin{minipage}[t]{4.5in}
  Specifies the number of {\tt TEST} operations, which have to be
  executed when clicking on \fbox{\tt M-TEST} button.
 \end{minipage}

\medskip

 \begin{minipage}[t]{0.7in}
 {\tt time}:
 \end{minipage}
 \ \hfill \
 \begin{minipage}[t]{4.5in}
  Sets the time counter to the given value.
 \end{minipage}
\end{minipage}

\bigskip

\begin{minipage}[t]{0.4in}
 \vphantom{x} \vspace{-4ex}
 \begin{picture}(22,40)(0,0)
  \put(11,20) {\circle{18}} \put(8,16){5}
 \end{picture}
\end{minipage}
 \ \hfill \ 
\begin{minipage}[t]{5.4in}
 The name of the file, in which the visualized data can be saved by
 activating the \fbox{\tt RECORD} button, can be specified here.
 The filename will be automatically extended by the suffix '.rec'.
 To change the filename, the \fbox{\tt RECORD} button must not be 
 activated. 
\end{minipage}

\bigskip

When the setup is left by clicking on \fbox{\tt CANCEL} all the changes 
made in the setup are lost. When leaving the setup by pressing the
\fbox{\tt DONE} button, the changes will be accepted if no errors 
could be detected.


\subsection{The Display Control Window of the Network Analyzer}

The display control window appears, when clicking on 
\fbox{\tt D-CTRL} button on the right side of the network analyzer 
window. This windows is used to easily change the area in the display
of the network analyzer.

\begin{figure}[htb]
 \begin{center}
  \begin{minipage}[htb]{\textwidth}
   \screendump{0.7}{xgui_figs/na_disp_ctrl.ps}
  \end{minipage}
 \caption{The display control window of the network analyzer} 
 \label{na_disp_ctrl}
 \end{center}
\end{figure}

The buttons to change the range of the displayed area in 
horizontal direction have the following functions:

\framebox[1.2cm][c]{$1/2 \, \times$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  The length of the interval is to be bisected. The lower bound remains.  
 \end{minipage}

\medskip

\framebox[1.2cm][c]{$\vphantom{/}2 \, \times \vphantom{/}$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  The length of the interval is to be doubled. The lower bound remains.  
 \end{minipage}

\medskip

\framebox[1.2cm][c]{$\vphantom{/} < \vphantom{/}$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  Shifts the range to the left by the width of a grid column.  
 \end{minipage}

\medskip

\framebox[1.2cm][c]{$\vphantom{/} > \vphantom{/}$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  Shifts the range to the right by the width of a grid column.  
 \end{minipage}

\medskip

\framebox[1.2cm][c]{$\vphantom{/} \ll \vphantom{/}$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  Shifts the range to the left by the length of the interval.
 \end{minipage}

\medskip

\framebox[1.2cm][c]{$\vphantom{/} \gg \vphantom{/}$} \quad : \hfill 
 \begin{minipage}[t]{5in}
  Shifts the range to the right by the length of the interval.
 \end{minipage}

\medskip

The buttons to change the range in vertical direction
have a corresponding function. To close the display control window, press
the \fbox{\tt DONE} button.

