Evolution of Language Models

What are Language Models?

Evolution of Language Models

1. Rule-based LM

2. Statistical LM

3. Neural LM

4. Pre-Trained LM

5. Large Language Models (LLM)

Rule-based Language Models

• Grammatical rules of a specific language were used to predict the next word in a sentence.

• For e.g. in English `I` will be followed by `am` not `are`, and `They` can be followed by `have` or `are` like these grammatical rules.

• However, there are many exceptions and it is quite difficult to handle all the rules of a language.

Statistical Language Models

• In this method, a large set of texts was analyzed and the word-level probability of a word after a set of words was determined in a statistical way.

• How many times does `am` appear after `I` that probability is compared with other words like `are` or `is`.

• In an advanced SLM n-gram models were used where instead of finding probability from a previous single word, the last bi-gram (two words), and tri-grams (three words) were used to find the probability of the next word.

• However, In English, a single word can have multiple meanings based on the context of the sentence. SLM can not able to determine the context of the sentence.

Neural Language Models

• With Word2Vec (Word to Vector), these models calculate the probability of the following words by neural networks. Example: RNN (Recurrent Neural Network), LSTM (Long Short Term Memory)

Pre-Trained Language Models

• With ELMo (Context-aware Word Embedding) and Self-Attention through Transformer raised the performance bar of NLP tasks. Example: BERT and GPT-2

Large Language Models (LLM)

• There is a thin line between PLM and LLM.

• Scaling model size and training data size of PLMs new emergent abilities of model discovered. Example: ChatGPT, LLaMA, Claude

• LLM is different from PLM broadly in three ways:
• Emergent abilities
• Prompting/Conversational Interface
• To attend the scale, Engineering and Research problems must be solved.
 

References