% papers/neuro/nips91w/summary.tex

% see johnw.t

\documentstyle[11pt]{article}

\setlength{\topmargin}{-0.5in}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textheight}{9.0in}
\setlength{\textwidth}{6.5in}
\setlength{\parindent}{0in}
\setlength{\parskip}{0.1in}

\begin{document}
\bibliographystyle{plain}

\title{NIPS*91 VLSI Workshop Summary}

\author{Organized by \\
Clifford Lau, Office of Naval Research \\
Jim Burr, Stanford University
}

\maketitle

\section{Introduction}

This is a report on the NIPS Post-Conference Workshop on VLSI Neural
Networks and Neurocomputers, which was held at Vail, CO on December 6-7,
1991.
\footnote{
This summary may be obtained in latex (.tex) or postscript (.ps) format
via anonymous ftp from

{\sf archive.cis.ohio-state.edu:/pub/neuroprose/NIPS91/vlsi.summary.\{tex,ps\}
}.
}
The objective of the two day workshop was to address the latest advances in
VLSI implementation of neural nets, and the design of high performance
neurocomputers.  The following individuals participated in the workshop:

\begin{center}
\begin{tabular}{ll}
Clifford Lau	& clau@charm.isi.edu		\\
Jim Burr	& burr@mojave.stanford.edu	\\
Pat Shoemaker	& shoe@cod.nosc.mil		\\
Alice Chiang	& alice@micro.ll.mit.edu	\\
Tony Montalvo	& tonym@sting.ece.ncsu.edu	\\
Shaolin Li	& shaolin@helmholtz.sdsc.edu	\\
Nathan Zeldes	& nzeldes@fab8.intel.com	\\
Torsten Lehmann	& torsten@eiffel.ei.dth.dk	\\
Michael Chuang	& chuang@micro.ll.mit.edu	\\
Hans Peter Graf	& hpg@neural.att.com		\\
Andreas Andreou & andreou@hermes.ece.jhu.edu	\\
Eduard Sackinger& edi@neural.att.com		\\
Josh Alspector	& josh@bellcore.com		\\
Fin Martin	& fmartin@sc9.intel.com		\\
Chin Park	& cpark@td2cad.intel.com	\\
Tony Jayakumar	& tony@flash.bellcore.com	\\
Ron Benson	& benson@hope.caltech.edu	\\
Marwan Jabri 	& marwan@ee.su.oz.au		\\
Hal McCartor	& hal@asi.com			\\
Alan Cooley 	& cooley@beno.css.gov		\\
John Wawrzynek	& johnw@cs.berkeley.edu		\\
Tom Schwartz	& -				\\
Steve Anderson	& -				\\
\end{tabular}
\end{center}

The following people did not attend the workshop but have requested copies
of the workshop summary:

\begin{center}
\begin{tabular}{ll}
Kelvin Wagner	& kelvin@boulder.colorado.edu	\\
Max van Daalen	& max@cs.rhbnc.ac.uk		\\
\end{tabular}
\end{center}

\section{Issues for discussion}

The plan was to address the following questions:

\begin{itemize}

\item How successful have VLSI implementations of neural nets been so far?
\item What high performance neural net chips are available today?
\item How are neural net chips different from DSP chips?
\item Are dedicated neural net chips being used in real applications?
\item What algorithms have been implemented?  Which ones have not been?  Why
      not?
\item How important is on-chip learning?
\item How much arithmetic precision is necessary?
\item Which is more important, capacity or performance?
\item What are the issues in constructing very large networks?  What are the
      scaling limits?
\item What is the appropriate architecture for high performance neurocomputers?
\item What are the input/output devices for such neurocomputers?
\item Can we achieve the equivalent of one TERAOPS sustained performance? How?
\item What are the application areas for supercomputing neurocomputers?
\item What application and programming support environment is necessary?
\end{itemize}

\section{Workshop agenda}

\begin{itemize}
\item Fri morning

	\begin{itemize}
	\item New chips
	\item Application survey: computation and capacity requirements
	\item New neuromorphic circuits and structures (not discussed)
	\item Synaptic cell area
	\item Connection energy: decision and storage (not discussed)
	\item Analog vs digital vs optical (not discussed)
	\end{itemize}

\item Fri evening

	\begin{itemize}
	\item case studies
	\begin{itemize}
	\item[] \begin{tabular}{ll}
	ASI CNAPS & Hal McCartor, Adaptive Solutions		\\
	RAP	  & John Wawrzynek, U.C.Berkeley and ICSI	\\
	CLC	  & Josh Alspector, Bellcore			\\
	LL CCD	  & Alice Chiang, MIT				\\
	\end{tabular}
	\end{itemize}
	\end{itemize}

\item Sat morning
	\begin{itemize}
	\item High performance neurocomputer attributes
	\item TeraOps processor design
	\end{itemize}

\item Sat evening
	\begin{itemize}
	\item Benchmarks
	\item CAD design tools
	\item Programming support environment
	\end{itemize}

\end{itemize}

\section{Chip and board level design issues}

As indicated in the agenda, discussions on Friday centered on chip level and
board level design issues.  During the opening discussions, it was clear that
VLSI neural net chip design is a series of trade-offs.  At the device level,
there are many possible ways and technologies to realize modifiable synapses
and the sigmoidal nonlinearity.  Device level issues were not intended to be
a part of the workshop.  At the chip and board level, however, there are
issues such as special purpose chips versus general purpose chips, network
size, expandability, and cascadability of the chips.  At the small system
level and high performance neurocomputer level, there are issues such bus
interfaces, software support, commercial and scientific applications, and
research environment.

Friday morning, we updated our list of reported neurochips, added a few
applications to the DARPA CPS/connections graph, and presented and updated a
table of synaptic cell areas. Friday evening, we heard from a number of
researchers who have implemented large network machines.

\section{New chips}

\begin{table}
\begin{center}

\scriptsize

\begin{tabular}{|l|l|l|c|c|c|c|c|c|c|c|c|}
\hline
who	& ref
	& type
	& tech
	& capacity
	& accuracy	% new
	& speed
	& power
	& l & p & c & a
					\\
	&		% ref
	&		% type
	&		% tech
	&  (N, S)	% capacity
	&  (N, S)	% accuracy
	&   CPS		% speed
	&chip, syn	% power
	& & & & 	% lpca
					\\
\hline
\hline
&&&&&&&&&&&\\

% new chips

SNAP			& TJS &&&&&&&&&&
\\
Chiang2			& -
			& CCD
			& 3$\mu$ CMOS
			& 192, 6K
			& 8-10, 6
			& 1.9e9
			& 8W, 4nJ
			& n&n&y&y
\\
SAIC			& TJS &&&&&&&&&&
\\
Motorola		& TJS &&&&&&&&&&
\\
Intel/Nestor		& TJS &&&&&&&&&&
\\
Synaptics		& TJS &&&&&&&&&&
\\
Siemens MA16		& \cite{Ram91}
			& digital
			& 1.0$\mu$ CMOS
			& off-chip DRAM
			& 16,16
			& 7.2e8
			& -
			& y&n&y&n
\\
Master Perturbator	& MJ &&&&&&&&&&
\\
Bourke			& MJ &&&&&&&&&&
\\
ANNA       & \cite{BS91}
           & D,A
           & 0.9$\mu$ CMOS
           & 64, 4096
           & 3, 6
           & 1e10
           & 0.75W, 75pJ
           & n&y&y&y
\\

&&&&&&&&&&&\\
\hline
&&&&&&&&&&&\\

% analog neurochips

%1
Caltech    & \cite{Agr90}
	   & CCD
	   & 2$\mu$ CMOS
	   & 256, 64K
	   & -
	   & 5e8
	   & -
           & n&n&-&-
\\
%2
Chiang	   & \cite{Chi90}
	   & CCD
	   & 3$\mu$ CMOS
	   & 144, 2016
	   & 8-10,6
	   & 1.44e9
	   & 2W, 1.4nJ
	   & n&n&y&y
\\
%3
USC	   & \cite{Lee90}
	   & cap
	   & 2$\mu$ CMOS
	   & 25, 2525
	   & -
	   & -
           & -
           & n&n&n&-
\\
%4
Matsushita & \cite{Mor90}
	   & analog
	   & 2.2$\mu$ BiCMOS
	   & 64, 768
	   & -
           & 7.7e7
	   & -
	   & n&n&n&-
\\
%5
ETANN	   & \cite{Ho89}
	   & analog
	   & 1.0$\mu$ EEPROM
	   & 64, 8192
	   & 8,8
	   & 8e9
	   & 2W, 250 pJ
	   & n&n&y&y
\\
%6
NET32K	   & \cite{Graf90}
	   & D,A
	   & 0.9$\mu$ CMOS
	   & 1024, 32K
	   & -
	   & 3e11
	   & 160mW, 500fJ
	   & n&n&y&-
\\
%7
ECLNN(CLC) & \cite{Al90}
	   & D,A
	   & 1.2$\mu$ CMOS
	   & 32, 1000
	   & ?,5
	   & 1e9
	   & 1W,1nJ
	   & y&n&y&-
\\
Mitsubishi & \cite{Ar90}
	   & analog
	   & 1.0u CMOS
	   & 125, 10K
	   & -
	   & 1e11
	   & 1.5W, 15pJ
	   & y&n&y&-
\\
%9
JPL	   & \cite{Eb89}
	   & analog
	   & 2.0$\mu$ CMOS
	   & 32, 1000
	   & -
	   & 1e9
	   & -
	   & n&n&n&-
\\

&&&&&&&&&&&\\
\hline
&&&&&&&&&&&\\

% digital neurochips

%10
CNAPS(X1)  & \cite{Ham90}
	   & digital
	   & 0.8$\mu$ CMOS
	   & -, 256K
	   & 8, 16
	   & 1e9
	   & 4W, 4nJ
	   & y&y&y&y
\\
%11
Ouali	   & \cite{OS90}
	   & digital
	   & 2.0$\mu$ CMOS
	   & -
	   & -
	   & -
	   & -
	   & n&n&y&-
\\
%12
Duranton   & \cite{DS89}
	   & digital
	   & 1.6$\mu$ CMOS
	   & -
	   & -
	   & 8e8
	   & -
	   & y&-&y&-
\\
%13
Hirai	   & \cite{Hir89}
	   & digital
	   & 1.2$\mu$ CMOS
	   & 6, 84
	   & -
	   & 8.4e4
	   & -
	   & n&n&y&-
\\
%14
DNNA	   & \cite{Tom90}
	   & stochastic
	   & -
	   & 32, 1024
	   & -
	   & 2e8
	   & -
	   & n&n&y&-
\\
%15
STONN	   & \cite{Wike90}
	   & stochastic
	   & -
	   & 20, 2000
	   & -
	   & 5e8
	   & -
	   & n&n&y&-
\\
%16
TInMANN	   & \cite{BM90}
	   & stochastic
	   &  2.0$\mu$ CMOS
	   & 1, ?
	   & -
	   & -
	   & -
	   & y&n&y&-
\\

&&&&&&&&&&&\\
\hline
&&&&&&&&&&&\\

% Waferscale nets

%17
Hitachi	   & \cite{Yas90}
	   & digital wsi
	   & 0.8$\mu$ CMOS
	   & 540, 34K
	   & -
	   & 1.4e8
	   & -
	   & n&n&y&-
\\

&&&&&&&&&&&\\
\hline
&&&&&&&&&&&\\

% neurocomputers and massively parallel processors

%18
CM1	   & \cite{BR87}
	   & mpp
	   & -
	   & -
	   & -
	   & 3e6
	   & -
	   & y&y&-&y
\\
%19
Warp	   & \cite{Po88}
	   & systolic
	   & -
	   & -
	   & -
	   & 17e6
	   & -
	   & y&y&-&y
\\
%20
AAP-2	   & \cite{Wat89}
	   & array
	   & -
	   & -
	   & -
	   & 18e6
	   & -
	   & y&y&-&y
\\
%21
SPRINT	   & \cite{Gro89}
	   & systolic
	   & -
	   & -, 1e6
	   & -
	   & 12e6
	   & 100W, 8$\mu$J
	   & y&y&-&y
\\

&&&&&&&&&&&\\
\hline
&&&&&&&&&&&\\

% biology

%22
neuron	   & \cite{DARPA88}
	   & -
	   & -
	   & 1e11,1e15
	   & 8 bit?
	   & 1e16
	   & 1W, 0.1fJ
	   & y&n&y&y
\\

&&&&&&&&&&&\\
\hline
\end
{tabular}

\normalsize

\end{center}
\caption{Summary of existing implementations. In the right four columns,
l = learning; p = programmable; c = cascadable; a = available.}
\label{tab:chips}
\end{table}

%----------

Table \ref{tab:chips} gives a summary of existing implementations. It is not
meant to be exhaustive; it is meant to be a sampling of a wide variety of
implementations. Analog, waferscale, neurocomputer, and biological entries
are for comparison purposes. The two numbers in the {\em power} column
are chip power dissipation and energy per connection.

\section{Application survey}

\begin{table}

\begin{center}
\begin{tabular}{|l|c|c|}
\hline
application & connections & CPS \\
\hline
\hline
regular edge detection		& 50K	& $10^9$		\\
image segmentation		& 20K	& $10^{10} - 10^{11}$	\\
check reading			& 20K	& 400M			\\
\hline
speech training			& 4M	& $10^{11}$ CUPS	\\
speech preprocessing		& 100K	& 200M			\\
speech recognition		& 40M	& $10^9$		\\
\hline
\end{tabular}
\end{center}

\caption{Capacity and performance requirements of recent applications.}
\label{tab:apps}
\end{table}

We reviewed the CPS vs connections graph published in the 1988 DARPA report
\cite{DARPA88} to see how well the application domains defined there
compared to capacity and performance requirements of recent applications.

We concluded the graph is still a good guide, with many of the more recent
applications falling within its domains.  A notable exception was vision,
where weight sharing has been used to greatly reduce the number of physical
connections required (see Table \ref{tab:apps}).

% should cite Hans Peter Graf's chips here - anyone have a reference?

% \section{Neuromorphic circuits and structures}
%
% Did not discuss.


\section{Synaptic cell area}

\begin{table}

\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
	  fets
	& type    
	& $\lambda\times\lambda$
	& $\lambda^2$
	& $\lambda^2/T$
	& rel.area
	& who	   \\
\hline
\hline
&&&&&&\\
  1T & EPROM   &   8$\times$8	 &    64 &  64   &   1(6)   & commercial   \\
&&&&&&\\
\hline
&&&&&&\\
  1T & DRAM    &   8$\times$8	 &    64 &  64   &   1(2)   &
						  \cite{Aok87}: 4 bit/cell \\
  1T & DRAM    &   8$\times$8	 &    64 &  64   &   1(6)   & commercial   \\
  4T & SRAM    &  12$\times$20	 &   240 &  60   &   4(24)  & commercial   \\
&&&&&&\\
\hline
&&&&&&\\
  1T & DRAM    &   8$\times$16	 &   128 & 128   &   2(12)  & MOSIS	   \\
  2T & DRAM    &  13$\times$31	 &   403 & 200   &   6(36)  & MOSIS	   \\
  3T & DRAM    &  24$\times$24	 &   576 & 192   &   9(54)  & MOSIS	   \\
  6T & SRAM    &  32$\times$36	 &  1152 & 192   &  18(108) & MOSIS	   \\
&&&&&&\\
\hline
&&&&&&\\
  7T & USC       &  40$\times$60   &  2400 & 342   &  37      & \cite{Lee90} \\
 12T & ETANN     &  83$\times$97   &  8036 & 671   & 125      & \cite{Ho89}  \\
 42T & Mitsubishi& 200$\times$200  & 40000 & 952   & 625      & \cite{Ar90}  \\
200T & ECLNN     & 160$\times$320  & 51200 & 256   & 700      & \cite{Al90}  \\
&&&&&&\\
\hline
&&&&&&\\
  1T & MIT CCD	 &  12$\times$10   &   120 & 120   &   2(12)  & MOSIS	     \\
  3T & MIT CAM	 &	-	   &    -  &  -    &   -      &	MIT	     \\
 30T & Andreas	 &  80$\times$80   &  6400 & 213   & 100      & JHU	     \\
 30T & Bourke	 & 106$\times$113  & 11978 & 399   & 187      & U.Sydney     \\
 30T & M.Pert    & 150$\times$180  & 27000 & 900   & 422      & U.Sydney     \\
&&&&&&\\
\hline
\end{tabular}
\end{center}

\caption{Area of various synaptic cells. Numbers in parentheses are
relative areas of 6-bit digital synapses.}
\label{tab:relarea}
\end{table}

We reviewed a table presented by Jim listing areas of various synaptic
cells, and added some new entries (see Table \ref{tab:relarea}).  This table
generated some lively discussion at the workshop. In a sense it is comparing
apples and oranges, because the digital synapses are just memory cells,
whereas most of the analog synapses include learning circuits. Also, the
digital cell area is quoted per bit whereas the analog cell stores the
weight to full, albeit limited, precision.

Despite these differences, the relative area occupied by a single synaptic
weight is an important parameter in computing the amount of silicon area
required for weight storage in an application with a known network capacity.

A conclusion which can be drawn from this table is that analog
implementations to date are better for low capacity, high performance
networks, and digital implementations are better for high capacity, lower
performance networks. This could change if someone were to build a
time-multiplexed analog network which amortized each learning circuit over a
cluster of synaptic weights.

% \section{Connection energy}
%
% Not discussed.

% \section{Analog vs digital vs optical}
%
% Not discussed.

\section{Case studies}

Friday afternoon, several researchers discussed specific implementations as
background material for Saturday morning's TeraOps processor design.

\subsection{ASI CNAPS}

Hal McCartor of Adaptive Solutions (hal@asi.com) discussed the CNAPS chip
and overall system design. The following is his contribution:

The current configuration for the CNAPS system is 4 CNAPS 1064 chips (64
processors per chip) on one board running at 20 MHz.  The processors are
fully programmable and operate as a massively parallel SIMD machine.  Each of
the 256 processors can perform a multiply and an add each clock cycle.

The system is supported by an assembler, a ``C'' compiler, a symbolic
debugger, and a graphical user interface.  There is support for calling the
hardware from a normal ``C'' program or a scripting language using algorithm
type, learning parameters, network size parameters etc. as arguments for the
call.



\subsection{RAP}

John Wawrzynek (johnw@cs.berkeley.edu) discussed the ICSI RAP.
The following is his contribution:

{\def\cite#1{[#1]}

The Ring Array Processor (RAP) was designed
for fast implementation of layered neural network
algorithms \cite{Beck, 1990; Bilmes and Kohn, 1990; Kohn and Bilmes,
1990; Morgan, 1990d; Morgan et al, 1990b}.
This system was implemented in roughly one year (from conception to
working prototype), and has since been in serious use for research
into connectionist applications, particularly speech recognition.
The RAP is a multi-DSP MIMD system
with a low-latency ring interconnection scheme
using programmable gate array technology and a significant amount
of local memory (16 MB) per node.
All inter-processor communication is handled by the ring.
Multiple 4-processor boards
serve as an array processor for a SPARC host.
Theoretical peak performance is 128 MFlops/board, and peak
ring communication speed is 64 MB/second/board. Test runs
with up to ten boards
(the largest configuration benchmarked)
show a sustained throughput of
roughly 10\% to 90\% of this for algorithms of
current interest.
For a net dimension of sufficient size, forward propagation
approaches 100\% efficiency (with respect to 64 million
multiply-accumulates/second/board for
a 16 MHz clock), and ten RAP boards are as much as
three orders of magnitude faster than a Sun SPARCStation-1 running the same
benchmark (typically 50-120 Million Connection Updates Per Second,
or MCUPS, and 200-570 Million Connections Per Second, or MCPS).
Software development is aided by
a Sun-based command interpreter,
a Unix-compatible library (e.g., fread() and fwrite()), and
an assembly language library of matrix and vector routines.

A reasonable design for a parallel system in which all processors
need values from all other processors is a single broadcast bus
(e.g., for forward propagation in a fully connected layered network).
However,
it is difficult to design a high-performance bus that
supports more than a handful of processors.
An alternative is a ring design, which is also more flexible for
other related algorithms such as the
backward phase of the
back-propagation learning algorithm, for which the
optimal storage of the weight matrix is in the opposite order.
To avoid duplication of the weight matrix to get both orderings,
one can use a circular
pipeline or ring architecture to distribute partial
sums to neighboring processors where local contributions
to these sums can be added.
When the processors must communicate with each other
through the ring, hardware synchronization automatically occurs.
A node that attempts to read before data is ready, or to write when
there is already data waiting, will stop executing until
the data can be moved.

The RAP DSP software is built in three levels. At the lowest
level are hand coded assembler routines for matrix, vector and ring
operations.
An intermediate level consists of matrix and vector object classes
coded in C++.
A programmer writing at this level or above can view the RAP
as a conventional serial machine.
These object classes divide the data and processing as evenly as possible
among the available processing nodes, using the ring
to redistribute data.
The top level of RAP software is the Connectionist Layered
Object-oriented NEtwork Simulator (CLONES) environment for
constructing ANNs \cite{Kohn 1990}.

}

\subsection{ECLNN(CLC)}

Josh Alspector (josh@bellcore.com) discussed his ECLNN chip. The following
is his contribution:

We report learning measurements from a system composed of a cascadable
learning chip, data generators and analyzers for training pattern
presentation, and an X-windows based software interface.  The 32 neuron
learning chip has 496 adaptive synapses and can perform Boltzmann and
mean-field learning using separate noise and gain controls.  We have used
this system to do learning experiments on the parity and replication
problem.  The system settling time limits the learning speed to about
100,000 patterns per second roughly independent of system size.


\subsection{LL CCD: CCD's for PATTERN RECOGNITION}

Alice Chiang (alice@micro.ll.mit.edu) discussed her CCD chips.
The following is her contribution:

     Recognition of a complex pattern in real time is an interesting,
useful and challenging goal.  One currently envisioned approach to this
pattern recognition problem uses a multilevel, hierarchical,
distributed computation and information neural network (NN)
architecture.  At the first processing level, processors will be used
for signal conditioning such as noise removal, enhancing and/or
skeletonizing an input image or Fourier transforming incoming speech.
At the next level, neural networks with local connections and
replicated weights are used to generate feature maps of the processed
input.  Finally, the extracted feature map will be passed to higher
level fully connected neural networks for object classification.  Each
of the above-mentioned processing level requires NN processors with
large numbers of neurons, interconnections and interconnection
computations/s, which poses very difficult VLSI design requirements,
especially if the system has limited power and weight constraints.
Since 1987, MIT Lincoln Laboratory has been developing high-speed,
low-power, CCD-based neural network modules based on a time-multiplexed
parallel-computation architecture.  These modules share common features
such as large numbers of time-multiplexed-parallel CCD computation
elements, digitally stored connection weights and serial I/O buffers to
support either multi-module ring-type or multi-level parallel,
pipelined system architecture.  In particular, CCD NN module consisting
of 224 neurons, providing 1.9 billion connections/s, 6K interconnection
storage has been developed and NN module consisting of 500 neurons,
providing 5 billion connections/s and 100K interconnection storage can
be developed in near future.


%----------

\section{System and neurocomputer issues}

The first discussion centered on the question, ``What are the attributes of
neurocomputers?''  The term ``neurocomputer'' refers to a computer that
mimics human information processing, with an architecture that is based on
the abstraction of the computational principles in the brain.  For that
reason, neurocomputers must be sufficiently different from conventional
computers to justify calling them by a different name.  The driving force
behind the need to design and and construct a high performance neurocomputer
is the fact that there is often a claim that a ``neural network chip can
outperform a Cray'' but yet never substantiated.  Therefore our goal is to
design a high performance neurocomputer with the equivalent of one TERAOPS,
to show that indeed a neurocomputer can exploit its massive parallelism to
achieve that kind of throughput rate.

If the neurocomputer were to mimic human information processing, it should
have some attributes that are similar to that of the human being, attributes
such as the ability to learn from experience, ability to remember things,
recall from partial information, recognition, reasoning, understanding, and
sensory motor data processing.  It must also be small size and low power.
It should have as a minimum the ability to process vision information and
languages.  The neurocomputer should not be constrained to the digital von
Neumann kind of architecture, but should be flexible and innovative.

However, since a language for that kind of innovative neurocomputer does not
yet exist, we are reduced to the current state of neurocomputers.  It is
felt that the following specifications, in the current terminology, would be
considered a supercomputing neurocomputer:

\begin{itemize}

\item $10^{12}$ connections per second
\item $2.5\times 10^{11}$ connection updates per second
\item $10^{9}$ connection weights (synapses)
\item 8 bit precision in numerical computations (16 bit preferred)
\item Small size
\item Low power
\item Low cost
\end{itemize}

We started on the design of such a high performance neurocomputer. It was
felt that the best digital architecture would be a mesh connected array.
Such a nearest neighbor interconnected mesh would reduce the massive
interconnections typically required in neural network realizations.
Furthermore, almost all neural networks can be mapped onto a mesh connected
array.  Such an array, because of its regular geometry and next neighbor
connections, is very expandable to larger size if the need arises.  The
processors would be neural network chips.  The following are estimates of
the number of chips required to build such a high performance neurocomputer:

\begin{enumerate}

\item RAP, ICSI

	\begin{itemize}
	\item TMS320 DSP chip as processor
	\item 16MCPS per processor
	\item 4 processors per board
	\item 64 MB per RAP board
	\item 0.25 CPS/C - too low
	\item Total 10,000 RAP boards
	\item 25 watts per board
	\item Total 250 KW power
	\end{itemize}

\item Intel i860
	\begin{itemize}
	\item 50MCPS per processor
	\item Size of the array would be roughly 140 x 140
	\item Total 20,000 chips
	\item 50 KB memory per node for 1000 CPS/C
	\item 40 KW power
	\end{itemize}

\item CNAPS, Adaptive Solutions, Inc.
	\begin{itemize}
	\item 25 MCPS per processor
	\item 4 KB per processor
	\item 6250 CPS/C - too high
	\item 64 processors per chip
	\item 4 chips per board
	\item Total 1,000 boards with 4000 chips for $10^9$ connections
	\item 16 KW power (4W/chip)
	\end{itemize}

\item Analog
	\begin{itemize}
	\item Mitsubishi Boltzmann chip:
	\item 300 neurons, 50K weights, $10^{11}$ CPS, 5W
	\item 2 MCPS/C - much too high
	\item $10^9$ connections require 20K chips, 100KW
	\item This gives over $10^{15}$ CPS.
	\end{itemize}

\item standard DRAMs
	\begin{itemize}
	\item 100ns access time
	\item 4 bits/cycle: 5 MCPS/chip
	\item 4 Mbit DRAM chips: 10 CPS/C - too low
	\item $10^{12}$ CPS requires 200,000 chips
	\item This gives $10^{11}$ connections
	\end{itemize}

\item page mode DRAMs
	\begin{itemize}
	\item Page mode access 100MHz
	\item 4 bits/cycle: 50 MCPS/chip
	\item 4 Mbit DRAM chips, page mode: 100 CPS/C - too low
	\item $10^{12}$ CPS requires 20,000 chips
	\item This gives $10^{10}$ connections
	\end{itemize}

\item high speed SRAMs
	\begin{itemize}
	\item 100MHz access, byte wide: 100 MCPS/chip
	\item 1 Mbit SRAM: 128K weights, 780 CPS/C - just about right
	\item $10^{12}$ CPS requires 10,000 chips
	\item This gives $10^{9}$ connections
	\end{itemize}

\item CCD, MIT Lincoln Lab.
	\begin{itemize}
	\item 500 KB per chip
	\item 50 processors per chip
	\item 20 MHz clock
	\item 2000 CPS/C - a little too high
	\item Total 2,000 chips
	\item 2 KW power
	\item This is beginning to look possible.
	\end{itemize}

\item MOSIS digital 0.8$\mu$ CMOS chip using 32$\times$32 SRAM (new design)
	\begin{itemize}
	\item 100 MHz clock
	\item 8 $\times$ 8 Multiply Accumulate
	\item 100 KB memory on-chip
	\item One processor per chip
	\item 1000 CPS/C - just right for this application
	\item Total 10,000 chips
	\item 1 KW at 5 volt power supply
	\item This looks very possible too.
	\end{itemize}

\item feedforward MOSIS analog
	\begin{itemize}
	\item Orbit 2.0$\mu$ double poly floating gates: 16$\times$16 $\lambda$
	\item 400K weights/chip
	\item 4e11 CPS/chip (400K weights every 1$\mu$sec)
	\item 1 MCPS/C - much too high
	\item 400mW/chip
	\item 2,500 chips for 10$^9$ connections
	\item 10$^{15}$ CPS
	\item 1KW at 5 volt
	\end{itemize}

\item Custom designed digital 0.8$\mu$ CMOS chip using 16$\times$16 SRAM
	\begin{itemize}
	\item 100 MHz clock
	\item 4 processors per chip, 8$\times$8 MAC
	\item 400 KB memory per chip
	\item 1000 CPS/C - just right
	\item 2,500 chips
	\item 1 KW at 5 volt, 40W at 1 volt (low voltage CMOS)
	\item This looks quite possible.
	\end{itemize}

\item Ideal feedforward 0.8$\mu$ analog
	\begin{itemize}
	\item 8$\times$8 $\lambda$ floating gates
	\item 10M weights/chip
	\item 1e13 CPS/chip (10M weights every 1$\mu$sec)
	\item 1 MCPS/C - much too high
	\item 10W/chip
	\item 100 chips for 10$^9$ connections
	\item 10$^{15}$ CPS
	\item 1KW at 5 volt
	\end{itemize}

\end{enumerate}

Although the chip design appears to be quite feasible, the workshop did not
discuss the system level issues associated with the architecture of such a
high performance neurocomputer.  Certainly a computer is more than just a
collection of chips.  The mesh connected array is only one possibility.
Furthermore, the issues of interconnection, communication protocol between
chips, and routing of the activations and/or weights were not discussed.
Many system integration issues remain open.


\section{Supercomputing neurocomputer applications}

In the afternoon session, discussion turned to application areas of such a
high performance neurocomputer.  Many potential applications were
identified, such as vision processing, language and speech processing,
weather modeling (which may require 16 bit precision), human genome
classification project, brain modeling, and even stock market analysis and
prediction.  However, since such a high performance machine currently does
not exist, it is premature to discuss its potential for application.
Undoubtedly if and when such a machine becomes a reality, new applications
will appear and be found to be in need of such a neurocomputer.

\section{Benchmarks}

Next, we turned to the problem of benchmarking for neurocomputers. The
rationale is that we want to be able to compare the performance of
neurocomputers so that the customers can objectively and intelligently
choose among available alternatives. Hopefully the benchmark should be a
problem that is independent of the neurocomputer realization, such as the
FFT in conventional computer benchmarks. Benchmarking turned out to be a
more difficult issue than it appears on the surface. Often it is difficult
to separate the problem from its solution. In other words, neural network
solutions are very often problem specific. It is generally agreed, however,
that benchmarks should be developed at the board level and not at the chip
level. This is because chips are only usable when they are put into a
board.

One possible benchmark is the problem of convolution, which is often
done in image processing. In this case, we can compare the time required to
perform a convolution with 16 x 16 kernel over a 512 x 512 pixel image.
Even in this case, the question arises as to whether the uploading and
downloading of the data should be a part of the benchmark. It is clear that
in order to have a reasonable set of benchmarks, it is necessary to develop
a set of performance criteria. In other words, what is it that we are
measuring and comparing?

Several possible performance parameters were discussed:  time to complete
task, accuracy of classification, size of training set, size of test set, \%
correct, \% error, time to train the net, time to make a decision,
processing throughput rate, cost, size, and power.  It was felt that most
likely the benchmarks should be involved in speech processing, vision, or
optical character recognition. However, it was felt that problems such
NETALK, ISOLET, and sonar were too small scale to really allow the power of
neural networks to shine. Other databases such as the NIST character set are
too difficult to use. In spite of the difficulty of identifying a set of
benchmarks, it was felt that benchmarks were necessary.

\subsection{Currently available benchmarks}

After the workshop, Scott Fahlman
({\sf Scott\_Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU})
sent Jim this email message in response to a query about accessing
Nettalk:

\begin{verbatim}

The data set that you want is part of the collection of learning benchmarks
that we are maintaining at CMU.

If you just want to access this collection via FTP, here are the general
instructions:

For people (at CMU, MIT, and soon some other places) with access to the
Andrew File System (AFS), you can access the files directly from directory
"/afs/cs.cmu.edu/project/connect/bench".  This file system uses the same
syntactic conventions as BSD Unix: case sensitive names, slashes for
subdirectories, no version numbers, etc.  The protection scheme is a bit
different, but that shouldn't matter to people just trying to read these
files.

For people accessing these files via FTP:

1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu".
The internet address of this machine is 128.2.254.155, for those who need it.

2. Log in as user "anonymous" with no password.  You may see an error
message that says "filenames may not have /.. in them" or something like
that.  Just ignore it.

3. Change remote directory to "/afs/cs/project/connect/bench".  Any
subdirectories of this one should also be accessible.  Parent directories
should not be.  NOTE: you must do this cd as one atomic operation, not try
to crawl in one directory at a time.

4. At this point FTP should be able to get a listing of files in this
directory and fetch the ones you want.

5. The directory "/afs/cs/project/connect/code" contains public-domain
programs implementing the Quickprop and Cascade-Correlation algorithms,
among other things.  Access it in the same way.

The exact FTP commands you use to change directories, list files, etc.,
will vary from one version of FTP to another.  I'm not sure if there's
currently any way to FTP files if you live on bitnet or csnet.  I've heard
for years about plans to add FTP capability between those nets and Arpanet,
but I don't think it has happened yet.  Once the collection is large
enough, we may try to find some organization willing to make tapes for
people at a reasonable cost.

Scott Fahlman
School of Computer Science
Carnegie Mellon University

\end{verbatim}


\section{Tools}

We spent the last half of the Saturday afternoon session discussing
tools for chip and network design.

\subsection{Chip design}

A number of public-domain or low-cost tools are available for chip
design. The following list is not meant to be exhaustive, but represents
tools actually used by one or more participants to design chips:

\begin{itemize}
\item Berkeley CAD suite
\begin{itemize}
\item access: send email to vlsi@ucbvax.berkeley.edu
\item {\sf magic} - VLSI layout editor
\item {\sf spice} - transistor level circuit simulation
\item {\sf irsim} - transistor level RC timing simulation (a descendant of
{\sf rsim} by Chris Terman of MIT, modified at Stanford, distributed with
the Berkeley tools).
\end{itemize}

\item Berkeley OCT tools
\begin{itemize}
\item access: send email to vlsi@ucbvax.berkeley.edu
\item {\sf misII} - logic synthesis
\item {\sf timberwolfe} - standard cell place and route
\end{itemize}

\item Berkeley Lager tools
\begin{itemize}
\item access: send email to vlsi@ucbvax.berkeley.edu
\item {\sf timlager} - tiling tool
\item {\sf flint} - interactive floorplanner
\end{itemize}

\item Caltech Analog suite
\begin{itemize}
\item access: anonymous ftp to hobiecat.cs.caltech.edu
\item {\sf analog} - device level simulator with nice user interface
\item {\sf wol} - layout editor
\item {\sf wolcomp} - layout composition tool
\item {\sf netcomp} - netlist comparator used to compare analog netlists with
	wol and wolcomp layouts
\end{itemize}

\item Tanner tools
\begin{itemize}
\item access: call (818) 795-2696
\item {\sf ledit} - layout editor, nice user interface, optimized for PCs
\item {\sf lvs} - layout vs schematic comparator
\item {\sf gatesim} - gate level simulator
\end{itemize}

\item misc tools
\begin{itemize}
\item access: ?
\item {\sf cande} - CCD simulation
\item {\sf CaZM} - spice rewrite from MCNC
\end{itemize}

\end{itemize}

\subsection{Network design}

We compiled a list of tools which participants have found useful in
constructing and simulating neural networks and evaluating their performance.
Two types of tools were identified: general and neural-net specific.

General tools include:

\begin{center}
\begin{tabular}{|l|l|l|}
\hline
	Tool		& source		& comments	\\
\hline
\hline
	Mathematica	& Wolfram Research	& (217) 398-0700 \\
	Matlab		& the MathWorks, Inc.	&		\\
	Labview		& 			& Mac only	\\
	Desire		&			&		\\
	CNet		& U. Washington		&		\\
\hline
\end{tabular}
\end{center}

Neural net tools include:

\begin{center}
\begin{tabular}{|l|l|l|}
\hline
	Tool		& source		& comments	\\
\hline
\hline
	NeuralWorks	& 			& Sun, Mac	\\
	Dynamind	&			& PC only	\\
	Neuralist	&			&		\\
	ICSIM		& ICSI			&		\\
	Aspirin,Migrane	&			&		\\
	Planet		&			&		\\
	Boolean		& U. Toronto		&		\\
	Cosmic		& NASA			&		\\
	Brainmaker	&			&		\\
	Neural shell	&			& PC only	\\
\hline
\end{tabular}
\end{center}


\section{Open issues}

Throughout the workshop, it was clear that there were a number of open
issues that needed resolution and discussion.  These open issues include:

\begin{itemize}

\item Mapping of neural network to processor architecture.

\item Mapping of the problem to the neural network approach (vice conventional
DSP approach).

\item Development of metric for measuring network performance.

\item Development of benchmarks.

\item Development of programming support environments (assembly language
programming is generally not acceptable, high level language such as C is
preferred.)

\item Development of sensors and actuators to realize the potential of neural
networks.

\item Manufacturability of large scale neural network chips, boards, and
systems.

\item Fault tolerance of large scale network.

\item Analog versus digital trade-offs, and design issues.

\item Development of I/O devices to exploit the advantage of neurocomputers.

\end{itemize}


\section{Summary}

Due to the limited time available for the workshop, it was not possible to
discuss in depth all the issues that we started out with.  We were overly
ambitious in terms of trying to come to resolution of many of the very
difficult issues.  

In terms of high performance neurocomputers, we found that it is quite
feasible to design a neurocomputer to achieve the equivalent of one TERAOPS.
Special custom design chips are most likely necessary.  Many of the
architecture and system integration issues remain unresolved, however.  Some
answers to the questions we set out in the beginning have been obtained, but
many questions remain.

It is generally felt that the workshop was vastly successful.  Many issues
were brought up concerning the VLSI realization of neural nets during lively
discussions. Everyone benefited from having participated in this workshop.

\section*{Application references}

\subsection*{ASI CNAPS}

\subsection*{RAP references}

Beck, J. (1990).
The Ring Array Processor (RAP): Hardware.
{\it International Computer Science Institute Technical Report TR-90-048}.

Bilmes, J., and Kohn, P., (1990).
Ring Array Processor (RAP): Software Architecture.
{\it International Computer Science Institute Technical Report TR-90-050}.

Kohn, P., (1991).
{\it CLONES: A Connectionist Layered Object-oriented NEtwork Simulator},
International Computer Science Institute, Technical Report In Prep.

Kohn, P. and Bilmes, J., (1990).
Ring Array Processor (RAP): Software User's Manual Version 1.0.
{\it International Computer Science Institute Technical Report TR-90-049}.

Morgan, N. (1990d).
The Ring Array Processor (RAP): Algorithms and Architectures.
{\it International Computer Science Institute Technical Report TR-90-047}.

Morgan, N., Beck, J., Kohn, P., Bilmes, J.,  Allman, E.,  and Beer, J.
(1990b).
The RAP: a Ring Array Processor for Layered Network Calculations.
{\it Proc. of Intl. Conf. on Application Specific Array Processors},
pp. 296-308.  IEEE Computer Society Press, Princeton, N.J.

\subsection*{ECLNN}

\subsection*{LL CCD}

\begin{thebibliography}{10}

\bibitem{Agr90}
Aharon~J. Agranat, Charles~F. Neugebauer, and Amnon Yariv.
\newblock {A CCD based neural network integrated circuit with 64K analog
  programmable synapses"}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:551--555, 1990.

\bibitem{Al90}
Joshua Alspector.
\newblock {CLC - A cascadeable learning chip}.
\newblock NIPS90 VLSI Workshop, December 1990.

\bibitem{Aok87}
Masakazu Aoki, Yoshinobu Nakagome, Masahi Horiguchi, Shin'ichi Ikenaga, and
  Katsuhiro Shimohigashi.
\newblock {A 16-level/cell dynamic memory}.
\newblock {\em IEEE Journal of Solid-State Circuits}, pages 297--299, April
  1987.

\bibitem{Ar90}
Yutaka Arima, Koichiro Mashiko, Keisuke Okada, Tsuyoshi Yamada, Atushi Maeda,
  Harufusa Kondoh, and Shinpei Kayano.
\newblock {A self-learning neural network chip with 125 neurons and 10K
  self-organization synapses}.
\newblock In {\em Symposium on VLSI Circuits}, pages 63--64, 1990.

\bibitem{BR87}
G.~Blelloch and C.~R. Rosenberg.
\newblock {Network learning on the Connection Machine}.
\newblock In {\em Proceedings of the Tenth International Joint Conference on
  Artificial Intelligence}, pages 323--326, 1987.
\newblock Milan, Italy.

\bibitem{BS91}
Bernhard Boser and Eduard Sackinger.
\newblock {An Analog Neural Network Processor with Programmable Topology}.
\newblock {\em IEEE Journal of Solid-State Circuits}, 26(12), December 1991.

\bibitem{Chi90}
Alice Chiang, Robert Mountain, James Reinold, Jeoffrey LaFranchise, James
  Gregory, and George Lincoln.
\newblock {A programmable CCD signal processor}.
\newblock In {\em IEEE International Solid-State Circuits Conference}, pages
  146--147, 1990.

\bibitem{BM90}
David~Van den Bout and Thomas K.~Miller III.
\newblock {TInMANN: The integer Markovian artificial neural network}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:205--211, 1989.

\bibitem{DS89}
M.~Duranton and J.A. Sirat.
\newblock {Learning on VLSI: A general purpose digital neurochip}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  1989.

\bibitem{Eb89}
Silvio Eberhardt, Tuan Duong, and Anil Thakoor.
\newblock {Design of parallel hardware neural network systems from custom
  analog VLSI `building block' chips}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:183--190, 1989.

\bibitem{Graf90}
Hans~Peter Graf and Don Henderson.
\newblock {A reconfigurable CMOS neural network}.
\newblock In {\em IEEE International Solid-State Circuits Conference}, pages
  144--145, 1990.

\bibitem{Gro89}
A.~J.~De Groot and S.~R. Parker.
\newblock {Systolic implementation of neural networks}.
\newblock In {\em SPIE, High Speed Computing II}, volume 1058, January 1989.

\bibitem{Ham90}
Dan Hammerstrom.
\newblock {A VLSI architecture for high-performance, low-cost, on-chip
  learning}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:537--544, 1990.

\bibitem{Hir89}
Yuzo Hirai, Katsuhiro Kamada, Minoru Yamada, and Mitsuo Ooyama.
\newblock {A digital neuro-chip with unlimited connectability for large scale
  neural networks}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:163--169, 1989.

\bibitem{Ho89}
Mark Holler, Simon Tam, Hernan Castro, and Ronald Benson.
\newblock {An electrically trainable artificial neural network (ETANN) with
  10240 ``floating gate'' synapses}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:191--196, 1989.

\bibitem{Tom90}
Max Stanford~Tomlinson Jr., Dennis~J. Walker, and Massimo~A. Sivilotti.
\newblock {A digital neural network architecture for VLSI}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:545--550, 1990.

\bibitem{Lee90}
Bang~W. Lee, Ji-Chien Lee, and Bing~J. Sheu.
\newblock {VLSI image processors using analog programmable synapses and
  neurons}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:575--580, 1990.

\bibitem{Mor90}
Takayuki Morishita, Youichi Tamura, and Tatsuo Otsuki.
\newblock {A BiCMOS analog neural network with dynamically updated weights}.
\newblock In {\em IEEE International Solid-State Circuits Conference}, pages
  142--143, 1990.

\bibitem{OS90}
J.~Ouali and G.~Saucier.
\newblock {Silicon compiler for neuro-ASICs}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:557--561, 1990.

\bibitem{Po88}
Dean~A. Pomerleau, George~L. Gusciora, David~S. Touretzky, and H.T. Kung.
\newblock {Neural network simulation at Warp speed: How we got 17 million
  connections per second}.
\newblock In {\em IEEE International Conference on Neural Networks}, pages
  II:143--150, 1988.

\bibitem{Ram91}
U.~Ramacher, J.~Beichter, W.~Raab, J.~Anlauf, N.~Bruls, M.~Hachmann, and
  M.~Wesseling.
\newblock {Design of a 1st generation neurocomputer}.
\newblock In U.~Ramacher and U.~Rucket, editors, {\em VLSI Design of Neural
  Networks}. Kluwer Academic, 1991.

\bibitem{Wat89}
Takumi Watanabe, Yoshi Sugiyama, Toshio Kondo, and Yoshihiro Kitamura.
\newblock {Neural network simulations on a massively parallel cellular array
  processor: AAP-2}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:155--161, 1989.

\bibitem{DARPA88}
Carol Weiszmann, editor.
\newblock {\em {DARPA Neural Network Study, October 1987 - February 1988}}.
\newblock AFCEA International Press, 1988.

\bibitem{Wike90}
William Wike, David~Van den Bout, and Thomas~Miller III.
\newblock {The VLSI implementation of STONN}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:593--598, 1990.

\bibitem{Yas90}
Moritoshi Yasunaga, Noboru Masuda, Masayoshi Yagyu, Mitsuo Asai, Minoru Yamada,
  and Akira Masaki.
\newblock {Design, fabrication and evaluation of a 5-inch wafer scale neural
  network LSI composed of 576 digital neurons}.
\newblock In {\em IJCNN International Joint Conference on Neural Networks},
  pages II:527--535, 1990.

\end{thebibliography}

\end{document}

alias nips91 \
clau@charm.isi.edu, \
burr@mojave.stanford.edu, \
shoe@cod.nosc.mil, \
alice@micro.ll.mit.edu, \
tonym@sting.ece.ncsu.edu, \
shaolin@helmholtz.sdsc.edu, \
nzeldes@fab8.intel.com, \
torsten@eiffel.ei.dth.dk, \
chuang@micro.ll.mit.edu, \
hpg@neural.att.com, \
andreou@hermes.ece.jhu.edu, \
edi@neural.att.com, \
josh@bellcore.com, \
fmartin@sc9.intel.com, \
cpark@td2cad.intel.com, \
tony@flash.bellcore.com, \
benson@hope.caltech.edu, \
marwan@ee.su.oz.au, \
hal@asi.com, \
cooley@beno.css.gov, \
johnw@cs.berkeley.edu, \
kelvin@boulder.colorado.edu, \
max@cs.rhbnc.ac.uk \
Scott_Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU
