Preskoči na glavno vsebino

Optimization for Multi Layer Perceptron: Without the Gradient


  These days, the publishing house Nova Publishers published the book, entitled Advances in Machine Learning Research. In it is a chapter entitled OPTIMIZATION FOR MULTI LAYER PERCEPTRON: WITHOUT THE GRADIENT where I describe two new algorithms for neural networks learning (Bipropagation and Border Pairs Method ). Both of them are much more powerful than their predecessors - Backpropagation algorithm. The second algorithm is among other things constructive.

Abstract of the book chapter


During the last twenty years, gradient-based methods have been primarily focused on the Feed Forward Artificial Neural Network learning field. They are the derivatives of Backpropagation method with various deficiencies. Some of these include an inability to: cluster and reduce noise, quantify data quality information, redundant learning data elimination. Other potential areas for improvement have been identified; including, random initialization of values of free parameters, dynamic learning from new data as it becomes available, explanation of states and settings in hidden layers of learned ANN, among other.
This chapter deals with the contemporary, non-gradient approach to ANN learning, which is not based on gradual reduction of remaining learning error anymore and it tries to eliminate most of the mentioned deficiencies. Introduction includes a chronological description of some methods, which deal with solutions of mentioned problems: Initializing Neural Networks using Decision Trees (Arunava Banerjee, 1994), DistAl: An inter-pattern distance-based constructive learning algorithm (Jihoon Yang, 1998), Geometrical synthesis of multilayer feedforward neural networks or Multi-Layer Perceptron (Rita Delogu, 2006) and Bipropagation - a new way of MLP learning (Bojan Ploj, 2009).We continue with the description of a new learning method - Border Pairs Method (BPM), which in comparison with the gradient methods carries numerous advantages or eliminates most of the predecessor’s deficiencies. The BMP implements and uses border pairs – learning patterns pairs in the input space, which are located close to the class border. 
The number of boundary pairs gives us some information about the complexity of the learning process. Boundary pairs are also the perfect basis for the noise reduction. We determine that performing a noise reduction of the border pairs is sufficient.
By dividing the input space, the homogenous areas (clusters) are established. For every linear segment of border we assign one neuron in the first layer. The MLP learning begins in the first layer by adapting individual neurons. Neurons on the first layers are saturated, so we get a binary code on the output of the first layer - the code is unified for all members of the same cluster. Logical operations based on the data from the first layer are executed in the following layers. Testing showed that such learning is reliable, it is not subject to overfitting, and is appropriate for on-line learning and susceptible to concept drift in the process of learning (forgetting and additional learning).

Komentarji

Priljubljene objave iz tega spletnega dnevnika

Artificial intelligence

          Artificial intelligence (AI) is a relatively young branch of science that stirs the imagination of many. Even movie directors from hollywood are not exceptions. Development in AI area is very fast and there is no indication that this will change soon. I still remember my first contact with learning devices. This happend at the end of the last millennium when I realized neural networks (NN). They have immediately attracted my attention, because such devices were not known till then.           NN are made along the lines of mammalian brain. During the learning NN extract the essence from the data. After the learning we can ask NN questions. It gives us the right answers even to questions that during learning did not participate.  NN learns autonomously and therefore may exceed the teacher's (poeple's) knowledge. Here are some important achievements of artificial intelligence: A couple of years ago the computer Deep Blue became the world champion of chess

Bionika, zanimiv poklic prihodnosti

Živimo v času, ko se znanje ustvarja in širi vedno hitreje in temu trendu se pridružujemo tudi izobraževalne ustanove. Ponudba smeri in stopenj izobraževalnih programov se vsako leto povečuje in med novimi programi prevladujejo interdisciplinarni. To ustreza manjšim podjetjem, ki so bolj prožna in odzivna in se zato lahko bolje znajdejo v tem gospodarsko razburkanem času. V malih podjetjih namreč ni prostora za množico ozko usmerjenih strokovnjakov, v prednosti so kadri z interdisciplinarno širino, ki jo omogočata mehatronika in predvsem bionika .      Na višji strokovni šoli na Ptuju smo že pred časom zaznali piš tega novega vetra in začeli prikrojevati naša jadra. Prvi korak v to smer je bil program Mehatronika , ki smo ga razvili pred približno desetletjem in povezuje med seboj več tehniških strok. Predvsem so to elektrotehnika, računalništvo in strojništvo. Trg delovne sile je potrdil našo vizijo saj se naši diplomanti uspešno zaposlujejo tako v Sloveniji, kakor tudi v t

Beyond Backpropagation

Gartner is predicting a very bright near future for the "Machine learning". 2015 was a peak year of inflated expectations, now, in 2016 is following period of disillusionment and in 2017 should be reached the plateau of productivity. Elsewhere this process usually last for 10 years. One kind of the most popular modern "machine learning" is named "Deep Learning" what is another name for neural networks with little bit more layers and perhaps even with a convolution and/or recursion. The learning of this kinds networks was until now usually based on gradient descent, on slow, iterative, non-reliable process named Backpropagation . That kind of learning is very demanding and extensive. On plain computer can last for hours or even many days and is often unsuccessful concluded. Recently are appeared two algorithms that significantly improve this kind of machine learning: " Bipropagation " and " Border pairs method ". Bipropagat