A neural network is a collection of "processing nodes" transferring activity to each other via connections [Tayl93]. The clearest example is, of course, the brain. Nerve cells fire impulses to each other across axons and detect the impulses through dendrites. The physical mechanisms of memory and learning remain a mystery, but we do know that people learn by practice and reinforcement, and that our knowledge can change as more information is received. An additional, important element of learning is the ability to infer knowledge, to make assumptions based on what we know and to apply what we have learned in the past to problems and situations which may not be completely understood.
In computing, "neural network" refers to a class of models which simulate learning in order to assist us in detecting information, predicting outcomes, and making decisions. Neural nets have the "topology of a directed graph" [Tayl93], meaning that, like the structure of the brain, connections between nodes or neurons is one-way. A two-way relationship between nodes implies two, one-way connections. Connections between graph nodes are called edges; these are similar to axons and dendrites in the brain. One theory of the physiology of human learning by repetition is that the repeated sequences of impulses strengthen connections between neurons and form memory paths. To retrieve what was learned, nerve impulses follow these paths to the correct information. As we get "out of practice" these paths diminish and we forget what we have learned (click here for an illustration from [ Squi87]). In order for a machine to "learn," edges are placed from node to node, forming paths which lead to the right conclusions.
The old adage "Necessity is the mother of Invention" may be true, but Curiosity and Laziness are probably in the family somewhere. People have always sought better, easier and faster ways to do things, either because of a pressing need or purely for the satisfaction of discovery. Recall that many technological developments have been motivated either by military needs or the drive for automation and increased productivity in business. Essentially, we ask computers to do those things we already know how to do, but we want them to do it faster and and more accurately than we can. The human brain, with its 15 billion neurons, performs fantastically "at such tasks as vision, speech, information retrieval, and complex spatial and temporal pattern recognition in the presence of noisy or distorted data" [Kara93]. These are some of the things researchers are trying to train machines to do. Neural networks are a natural extension of exploring the limits of computing, in terms of methodology and theory. In terms of application, neural networks are found in systems performing image and signal processing, pattern recognition, robotics, automatic navigation, prediction and forecasting, and a variety of simulations. They are used to help humans wade through huge amounts of data and to extract relevant information. Neural nets allow the programming of systems to solve "problems in which the rules governing the situation are unknown or difficult to use" [Tayl93] and to apply computing to problems not solvable in a strictly linear fashion.
Neural networks, along with other analytical and predictive methods, are today being incorporated into a relatively new field called Knowledge Discovery and Data Mining (KDD). A primary goal of KDD tools is to assist the user in detecting relationships in data which may not be readily perceptible due to the sheer size of data, missing data elements, or certainly lack of time to examine data and infer knowledge from them. There are many applications of KDD tools in decision support and pattern recognition systems:
Artificial neural networks may be represented by different data structures, but they are each designed to "make use of some of the organizational principles felt to be used by the brain" [Stac94]. The neural net is composed of nodes named binary decision units (BDNs) by pioneers McCulloch and Pitts [Tayl93]. Each BDN has a number of inputs and a rule to emit activity (a "1") if the sum of the inputs is above a certain, threshold value and otherwise output no activity (a "0"). This is logically the same model as an individual nerve cell (neuron); hence these nodes are called neurons as well (see a picture from Grolier's The New Book of Knowledge encyclopedia, 1992). The neurons are connected into a directed graph, with a node's output feeding as input to another node in a tree-like heirarchy. In a "training" session, connections (edges in the graph) are weighted with numbers which multiply that input's value to give the desired effect on the BDN. A diagram is found in [Rowe94]. The neural net can be "feedforward" to create a decision path or "feedback" to allow back-propagation of error data to improve accuracy. Code for a linked list implementation is also found in [Rowe94]. Neural Nets may also be implemented in hardware for improved performance [Tayl93].
The behavior of the artificial neural net is determined, like the human brain, by the structure and strengths of the connections between neurons [Stac94]. And, much like human learning, the neural net must have good training in order to respond as desired. A neural network's learning is defined by [Stac94] to be "any change in the memory (weight matrix)." There are a variety of strategies to achieve training of the net and its continued learning. These may involve detailed applications of statical and numerical methods such as back-propagation of error, least-squares fitting, differential equations and other iterative methods. The general procedure is to introduce to the net some patterns or pre-processed data with identified features and compare the output to the desired response. Corrections are made to the edge weights and the process is repeated [Rowe94].
Three categories of learning are supervised , reinforced , and unsupervised [Tayl93]. In supervised learning or training, an input is given along with the desired repsonse and weights are adjusted to minimize the difference in output. Reinforcement training mimics B.F. Skinner's psychological model, using reward and penalty signals and modifying connection weights to maximize probability of receiving a reward while minimizing penalties. Reinforcement learning approximates optimization, one of the classic intractable computing problems. Neural nets using reinforcement algorithms are being used to recommend solutions to "unsolvable" problems like process planning, scheduling and "traveling salesman" problems in industry [Cheu94], [Posa94]. In unsupervised learning, the neural net itself detects the features and patterns in the input space and develops behaviors representing different classifications of inputs. These three general methods may be customized from training to ongoing learning to create a robust, dynamic network.
Technological advances enable us to collect far more information than we can possibly analyze, while our time to accomplish our goals seems to be vanishing in this fast-paced world. We need help from intelligent assistants, expert systems, data mining, and complex information retrieval strategies to make sense of it all. Intelligent applications, including those using neural networks, will be increasingly in demand as we suffer from workloads of more and more raw data and a highly competitive, global economic climate. According to [Kara93], the "neural network industry is expected to grow to some billions of dollars in annual U.S. sales by the year 2000 and then sustain an average annual growth rate of 40% each year."
