Computing 690

Assignment #1 Report

The Dendronic Learning Engine

Submitted by Bruce Matichuk and Shengjiu Wang

October 20, 1999

The Dendronic Learning Engine is based on the Adaptive Logical Network (ALN) machine learning technology invented by Bill Armstrong at the University of Alberta. Dendronic Decisions owns the technology and markets a software development kit (SDK) and a workbench application as part of the Dendronic Learning Engine. The SDK includes APIs in C and C++ that programmers can use to incorporate Adaptive Logical Networks into other applications. Embedded into an application, the learning engine can be used for supervised learning, on-line training, and reinforcement learning.

The ALNBench is a Windows application developed by Dendronic Systems Inc. ALNBench was designed as a tool for on-line testing and investigation of simple ALN models. ALNBench accepts a multi-column table of data with one column designated as the "output" column. The output column represents the values of the output function that is to be learned. Using a supervised learning approach, the user is able to train the ALN and to observe the predictive results of the learned ALN using a test data set.

ALNBench is a free program that illustrates many of the capabilities of the Dendronic Learning Engine. The test version software is easy to install and also easy to use. We did several tests with some sample data provided with the demonstration package, after training, ALNBench showed great accuracy, above 90%. The computation of Dendronic Learning Engine is also very fast. The use of continuous, piecewise linear approximations of functions, allows the engine to leave out large parts of a computation during analysis. Retrieving information from the dendronic tree generally takes a small number of lookups and is extremely fast. All the operations are spreadsheet-based, so data from any source can be easily input and analyzed within one user interface.

Data may be exported or imported using both the "export" and "import" menu items or by copying and pasting data from/to the training and test spreadsheets within ALNBench.

ALNBench provides an easy-to-use interface for manipulating training and test data, and for viewing results, the application has five views, from which the training results can be easily viewed. The views include: workspace, charts, report, LF statistics and log view. The Workspace view presents two spreadsheets along with other parameter alteration controls. The first spreadsheet is used to set up training data for the ALN. The second is used for test data. Initially when the ALN is trained there will be a high degree of error in the learned model since all of the linear pieces of the model cannot be made to fit at once. Over multiple training epochs the overall error of the model will go down. This progress can be monitored in the Charts view. The Reports view will provide a summary of the error results of each epoch. The number of linear pieces is also provided. The LF Statistics view provides statistical information on each linear piece created in the learned model. This view is only valuable to individuals who have a good understanding of ALN structure. The final view is the Log view in which a summary of learning results is given for each epoch.

Creating and training ALNs is a very simple process.

Step 1: Create an ALN with the appropriate number of data columns. One column must be designated as the "output" column. The output column will be the column that the ALN will attempt to learn.

Step 2: Paste or import data from a spreadsheet or tab deliminated file into the training spreadsheet.

Step 3: Select part of the test set to use as test data. This data will be used to validate the model.

Step 4: Run the training.

Step 5: Modify training parameters and re-run training as necessary until a model with sufficiently low error is generated.

The Adaptive Logical Network technology has been available as a commercial product for several years. Some uses include analysis, data prediction, classification and control.

Analysis:

The use of Adaptive Logic Networks for data analysis makes use of their ability to recognize patterns by extracting useful information from noisy data. As a data analysis tool, the results are used to provide classifications or a continuous real-valued output that is functionally related to the input variables.

Prediction and Forecasting Applications:

Prediction, or forecasting, is the ability of the system to predict future values or outcomes based on currently known input values. The approach is for the system to learn the relationship between the input and output variables from historical data presented during a learning phase.

Control Applications:

Control problems typically involve controlling one or more output quantities based on the state of several input quantities to a system. A control system is a well understood model that describes the relationship between the known input and the desired output quantities. When the relationship between the input and output is not well understood, making it difficult or impossible to create the required model, machine learning technology such ALNs may be used to learn the input-output relationship.

ALNs have been successfully used to solve many kinds of problems normally solved with neural nets. There are some distinct advantages with using ALNs instead of neural nets. The key advantages are listed below.

1. Safety

Safety and mission-critical applications require that system responses be understood for all possible inputs. Since an Adaptive Logic Network model is composed of linear surfaces that are well understood mathematically, proofs about the accuracy of an Adaptive Logic Network model for all inputs are feasible without requiring exhaustive testing.

2. Speed

Adaptive Logic Networks are very fast because the evaluation of a trained network typically involves simple comparison operations and a limited number of linear surface calculations. The speed of an Adaptive Logic Network lets it process large volumes of data on ordinary desktop computers.

3. Scalability

An Adaptive Logic Network can handle complex problems with many input variables. Instead of adapting a fixed internal architecture, an Adaptive Logic Network’s architecture can grow dynamically and efficiently in response to the complexity of the training data. The structure of an Adaptive Logic Network efficiently represents the relationship between your problem’s inputs and outputs.

4. Broad domain applicability

Adaptive Logic Networks have successful applications in many problem domains. An investment in Adaptive Logic Network technology can pay for itself repeatedly, greatly reducing the complexity of the tool set required to build intelligent applications.

5. Embedding expert knowledge

Learning systems often have difficulties when there is a lack of historical data for training, or data contains too much noise. Adaptive Logic Networks can compensate for these problems during the learning phase by constraining their internal structure. These constraints are often based on physical laws and rules of thumb that dictate that certain relationships in the data must hold. Capturing common sense or even expert knowledge of a problem domain can compensate for sparse and noisy data, often resulting in a faster learning phase.

6. Function inversion

The ability of a model to provide the required input given a desired output can be very useful in the real-time control of machines and processes. The process of exchanging the role of inputs and outputs, or function inversion, is facilitated by Adaptive Logic Networks because the comparison operations that combine linear surfaces in an Adaptive Logic Network preserve a mathematical property called monotonicity. This property permits the internal structure of an Adaptive Logic Network to be rearranged so that the output of the network effectively trades places with one of the inputs.

7. Ease of understanding

The learning phase of an Adaptive Logic Network is controlled by parameters directly related to the properties of the data. Users need only be familiar with their data, not with the way the learning algorithms work. The user can concentrate on solving the problem, rather than on becoming an expert in the underlying technology.

Other advantages include:

• Uses arbitrary ranges of data values.
• Grows the architecture automatically.
• Allows the user to easily determine how to set the parameters.
• Trains and evaluates at high speed.
• Tells the user the sensitivity of the output to each input in different parts of the input space.
• Gives the user unprecedented control over.

1. The elimination of nodes and weights that are not needed.
2. The increasing or decreasing nature of the learned function .
3. The general shape of the solution.
4. The bounds on rates of increase or decrease with respect to each input.

Although the Dendronic Learning Engine is powerful, there are several limitations. During training, the user cannot determine how to set the ALN parameters needed to achieve acceptable accuracy while avoiding over-training. There is no method for determining whether the available data is sufficient for training. There are no visualization tools provided with ALNBench.