current community

your communities

more stack exchange communities

Stack Exchange
tour help
stack overflow careers
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 1 down vote favorite

I'm trying to implement an Auto-encoder by my own in Java. From the theory, I understood that auto-encoder is basically a symmetric network.

So, if I chose to have 5 layers in total, do I have to use 9 layers in training (back propagation) phase or 5 layers enough?

I've been reading theory but they are too abstract and full of math formulas, I could not get any implementation details via google.

What's the usual way of doing this?

enter image description here

An Auto-encoder, in training phase, using back propagation, tries to get the output similar to the input with a goal to minimize the error. It is shown above. The number of layers in the above image are 7 while the actual layers are 4 after the training. So, while training can I implement the back-propagation with just 4? If so, how can I do this?

share|improve this question

closed as too broad by Sean Owen, Lance Roberts, Kevin Panko, Thomas Jungblut, laalto Dec 29 '13 at 9:05

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs. If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer 1

up vote 2 down vote

Simple backpropagation won't work with so many layers. Due to so called vanishing gradient pehenomen, networks having more than two hidden layers won't learn anything reasonable. In fact, best results are obtained with one hidden layer. So in case of autoencoder you should have INPUT layer, HIDDEN layer and OUTPUT layer. No need for more, the Universal Approximation Theorem clearly shows, that this is enough for any problem.

From the OOP point of view it depends whether you plan to reuse this code with different types of neurons, and by type of neuron I mean something deeper than just different activation function - different behaviour (stochastic neurons?); different topologies (not fully connected networks). If not - modeling each neuron as a separate object is completely redundant.

share|improve this answer
    
I know this....I'm planning to use RBMs for pretraining...Meanwhile, I'm doing this to get an idea of implementation..and I'm not sure if I have to use double the number of layers in training or just the right number of layers in (double because I have to get the input from input) –  pinkpanther Dec 28 '13 at 20:10
    
It does not really matter whether you use RBM or not - RBMs are also (efficiently) trainable only with 1 hidden layer. Training of deep autoencoder is not what you have painted. –  lejlot Dec 28 '13 at 20:59
    
Please clarify me about the implementation detail I asked..... but this paper of Geoffry Hinton suggests to use multiple layers... citeseerx.ist.psu.edu/viewdoc/… –  pinkpanther Dec 28 '13 at 21:03
    
Can you please tell me whether I should use in total seven layers when we actually need 4 layers after training? Refer to the last para of my question please.... –  pinkpanther Apr 16 '14 at 19:18
    
Your question lacks many aspects. What is the purpose of the network. How is it exactly trained. In general it seems very bizzare idea to "unfold" autoencoder despite its application. –  lejlot Apr 16 '14 at 19:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.