Small Language Models (SLM)

Small Language Models (SLM) are the mini version of Large language Models. SLMs have less number of parameters compare to LLMs. These models are designed to perform tasks such as sentiment analysis, text generation, etc. similar to LLM but with less number of parameters. The less parameters helps to improve to computational efficiency, accessibility and adaptability.

Some examples of SLM are

1. Llama 2 7B - Released by Meta in July'23, it has 7 billion parameter
2. Phi2 - 2.7 billion parameter model, developed by Microsoft
3. Stable Beluga 7B - Developed by Stability AI , it is auto regressive language model fine tuned on llam2
4. Xgen - Developed by Salesforce. It is a smaller scale model that is customized for particular domains.
5. Alibaba's Qwen - Developed by Alibaba cloud
6. Alpaca 7B - Fine tuned on Meta Llama 7B model on 52 K instruction-following demonstration.
7. Falcon 7B – This model is 7B parameters causal decoder-only model developed by TII and trained on 1,500B token.
8. MPT - Developed by MosaicML Foundation
9. Zephyr - Zephyr 7B is a model created by the HuggingFace H4 (Helpful, Honest, Harmless, Huggy) team with an objective to create a smaller language model that is aligned with user intent and outperforms even bigger models.
10. MobileBERT
11. GPT-Neo and GPT-J
12. T5-Small - Text-To-Text Transfer Transformer (T5) is a pre-trained encoder-decoder model handling all NLP tasks as a unified text-to-text-format where the input and output are always text strings. T5-Small is the checkpoint with 60 million parameters.