Understanding Layers in Recurrent Neural Networks for NLP

Question

In convolution neural networks, we have a concept that inner layers learn fine features like lines and edges, while outer layers learn more complex shapes.

Do we have any such understanding for layers in RNNs (like LSTMs), something like inner layers understand grammar while outer layers understand more complete meanings of sentences assuming that we are using the LSTM for some natural language task like text summarization?

ashutosh singh · Answer

Its not like it just understands grammar.

In LSTMs the network tries to preserve the hidden states over time. By doing this they try to learn long-term dependencies in the language and relationships between words at variable distances.

LSTM does this by using its three famous gates.

Forget gate -  Tries to remember only the important features and relationships overtime.
Input gate - Adds new information to old cell state at each time time step.
Output gate - Produces new output by taking into account old cell state and output at each time step.

Ta_Req · Answer

RNN/LSTM is designed for series (data has time step) like data(E. g. a sentence ) which has dependency between different parts of the data. In English, some words in a sentence have a dependency on previous words. 
To carry the dependency information and ignore the non-important information until the end of the sentence  RNN/LSTM was introduced.

If you use other variants of deep neural network (MLP) in series like data that network the network forget dependency information.

Understanding Layers in Recurrent Neural Networks for NLP

2 Answers

Add your own answers!

Ask a Question