Embed, encode, attend, predict - Dr. Matthew Honnibal

Embed, encode, attend, predict: A four step framework for understanding neural network approaches

While there is a wide literature on developing neural networks for natural language understanding, the networks all have the same general architecture, determined by basic facts about the nature of linguistic input. In this talk I name and explain the four components (embed, encode, attend, predict), give a brief history of approaches to each subproblem, and explain two sophisticated networks in terms of this framework -- one for text classification, and another for textual entailment. The talk assumes a general knowledge of neural networks and machine learning. The talk should be especially suitable for people who have been working on computer vision or other problems.

Just as computer vision models are designed around the fact that images are two or three-dimensional arrays of continuous values, NLP models are designed around the fact that text is a linear sequence of discrete symbols that form a hierarchical structure: letters are grouped into words, which are grouped into larger syntactic units (phrases, clauses, etc), which are grouped into larger discursive structures (utterances, paragraphs, sections, etc).

Because the input symbols are discrete (letters, words, etc), the first step is "embed": map the discrete symbols into continuous vector representations. Because the input is a sequence, the second step is "encode": update the vector representation for each symbol given the surrounding context. You can't understand a sentence by looking up each word in the dictionary --- context matters. Because the input is hierarchical, sentences mean more than the sum of their parts. This motivates step three, attend: learn a further mapping from a variable-length matrix to a fixed-width vector, which we can then use to predict some specific information about the meaning of the text.