The "Clamp and Release" (CAR) Approach
CAR was implemented to minimize the "monopolistic" tendency of the most active connection weights of the ANN during training. To achieve this goal, the values of the connections of the most active units have to be inhibited (clamped) from changing after reaching a predetermined deviation limit. Once these values are clamped, the error can just be minimized through the changes of the values of the less active unclamped units. Obviously, the number of clamped connection weights increases as the training proceeds as more units will reach the deviation limit. Definitely, learning will be significantly retarded if majority of the units will already be clamped. A limit then has to be established as to how many connections are to be clamped. Once this limit is reached, clamping will be stopped and all the clamped connections will be released. Figure 1 shows an example of the clamping sequence of the ANN units connections during training.
|
0
iteration
0% clamped |
15
iteration
9.1% clamped |
26
iteration
16% clamped |
38
iteration
25% clamped |
38
iteration
0% clamped |
|
Figure 1. The clamp and release sequence during training of ANN with 25 percent
clamped connection limit.
Connection Weights Deviation and Clamped Connections limits
CAR can just be implemented if the connection weights deviation and clamped connections limits are earlier determined. In this study, a series of experiments were conducted to determine the two important parameters to be used for the IBL. The values of the deviation limits used in this experiment, expressed in percent change of the initial values of the connection weights ranges from 25% to 300% at 25 % interval. On the other hand, the values of the clamped connection limits, expressed as percentage of the total number of connections of the ANN, ranges from 5% to 90% at 5% interval.
A Landsat TM data covering an area of 230 km2 was used for this experiment. The study area was located in Nishinasuno, Tochigi Prefecture, Japan. Training and test pixels were obtained through actual field visit of the study area. Seven land cover classes were considered in this study and they were the following: 1.) buildup areas; 2.) grassland; 3.) deciduous forest; 4.) coniferous forest; 5.) rice field; 6.) river and ; 7.) pond/dam.
Artificial Neural Network Structure
A multi-layer perceptron trained by the back-propagation algorithm (Rumelhart et al., 1986) using the gradient descent training method was used in this experiment. The ANN has one hidden layer with 30 nodes. The input layer has 5 nodes used to accommodate the 5 TM bands (bands 1 to 5) and the output layer consisted of 7 nodes reflecting the 7 land cover classes. For comparison purposes, two ANN of different sizes were trained using the conventional method and served as controls. The first ANN (control a) is a small network with 15 nodes in the middle layer and the second one (control b) has the same size as the networks trained using IBL. In this study, all ANN training were stopped after the trained networks reached an accuracy of at least 90% using the training data.
Results And Discussion
Training Speed
The results of the different training runs to determine the best combination of the deviation and clamped percentage limits that can give the fastest training time is shown in figure 2. Each line in figure 2 represents a deviation limit and shows the change in the number of iterations as the percentage of clamped connection increases. The two straight horizontal lines represent the number of iterations using the conventional training method (controls). It is very evident from the graph that most of the points are below the horizontal lines indicating that the IBL method results to faster training speed compared to the conventional one. Table 1 shows the fastest training speed of 2417 iterations (54.4 seconds) using the connection weight deviation and clamped connection limits of 225% and 15%, respectively. The conventional training method required 8239 iterations (101.6 seconds) and 6495 iterations (148.8 seconds) for the small ANN (Control a) and big ANN (Control b), respectively. The results also show that using the IBL, large ANN learn faster compared to the smaller ANN trained using the conventional method.
The effect of the clamped connection percentage is very evident in the experiment. The results show that training speeds were faster at clamped connection limits ranging from 15 to 45 percent. On the other hand, training speed did not significantly change when the clamped connection limits were 10 or lower at all levels of deviation limits. This indicates that only few active connections were identified at this level hence the effect on the training speed was minimal. On the other hand, training speed started to significantly decrease as the number of clamped connections goes beyond 85 percent. This result suggests that once the number of clamped connections reaches a critical mass, learning retardation will become very significant for only very few unclamped connections will be left to change to reduce the error.
The fastest training speeds were obtained at higher deviation limits and lower clamped connection limits. As indicated in figure 2, deviation limits ranging from 175 to 250 percent with clamped connection limits values ranging from 15 to 20 percent generated the best result. On the other hand, lower deviation limits generally yielded longer training time. This is because lower deviation limit will make connection weights that changes slightly qualify to be clamped. Consequently, more units will be clamped at the early stage of training which result to the significant retardation of the neural network to learn. On the other hand, higher deviation limits just allow connection weights that change significantly faster than most connection to be clamped. Because of this, the combination of higher deviation and lower clamped connection limits is sufficient to inhibit the changes of the "monopolistic" few connections in the ANN while giving more time for the less active units to participate in the error reduction process.
Figure 2. ANN training speed using different weight deviation and clamped
connection limits values.
Accuracy
Artificial neural network accuracy is always related to its ability to generalize. Generalization is the ability of the neural network to interpolate and extrapolate data that it has not seen before (Atkinson and Tatnall, 1997). One of the most important factors that effect the ANN generalization capability is training time. The longer a network is trained the better it classifies the training data, but the more likely it is to fail in the classification of new data because it may become over-specific (Kavzoglu and Mather, 1999). In this study, the combination of deviation limits and clamped percentage limits that resulted to the shortest training time were determined. The accuracy of the trained networks obtained from these limit settings were investigated whether the shortened training time also resulted to the improvement of the network's ability to generalize.
Table 1 shows the combination of deviation and clamped connection limits that resulted to the fastest training speeds in the experiment. The training speeds and accuracy obtained using the conventional training method are also included for comparison. The results show that IBL method with deviation limits ranging from 175 to 250 percent with 15 % clamped percentage limit resulted to training speeds more than twice as fast as the controls. It also shows that using the IBL, the ANN's accuracy to identify test pixels is greater than those trained using the conventional method. These results indicate that with shortened training time, the ability of the ANN to generalize also improves. Furthermore, the large ANN trained using the IBL can generalize better than the small ANN trained using the conventional method. Figure 3 shows two classified images using trained ANN employing the conventional and the IBL training methods.
Table 1. Trained ANN accuracy using the combinations of weight deviation and clamped connection limits that resulted to the fastest training speeds.
| Weight Deviation Limit (%) |
Clamped Connection Limit (%) |
Number of iterations |
Training Time (seconds) |
Accuracy (%) |
| control a (small ANN) |
control a |
8239 |
101.6 |
82.0 |
| control b |
control b |
6495 |
148.8 |
88.3 |
| 175 |
15 |
3135 |
75.5 |
89.1 |
| 200 |
15 |
2893 |
66.1 |
89.4 |
| 225 |
15 |
2417 |
54.4 |
90.0 |
| 250 |
15 |
2670 |
59.9 |
89.4 |
