The Rise of Small Language Models in AI

In the AI conflict, where tech giants have been racing to create ever-larger language models, an unexpected new trend has emerged: small is the new huge. As development in large language models (LLMs) appears to be plateauing, researchers and developers are increasingly focusing on small language models (SLMs). These tiny, efficient, and highly adaptive AI models challenge the concept that bigger is always better, with the potential to revolutionize the way we approach AI research.

Are LLMs beginning to plateau?

Recent performance comparisons released by Vellum and HuggingFace indicate that the performance gap between LLMs is rapidly closing. This tendency is especially noticeable in tasks like multiple-choice questions, reasoning, and math problems, where the performance disparities between the top models are minor. For example, in multi-choice questions, Claude 3 Opus, GPT-4, and Gemini Ultra all score above 83%, yet in reasoning problems, Claude 3 Opus, GPT-4, and Gemini 1.5 Pro all reach 92%.

Interestingly, smaller models such as Mixtral 8x7B and Llama 2 – 70B outperform bigger models in certain areas, such as reasoning and multi-choice problems. This shows that model size may not be the only determining factor in performance, and that architecture, training data, and fine-tuning strategies may all play an important role.

The most recent research publications introducing new LLMs all lead in the same direction: “If you just look empirically, the last dozen or so articles that have come out, they’re kind of all in the same general territory as GPT-4,” says Gary Marcus, former head of Uber AI and author of “Rebooting AI,” a book about developing trustworthy AI. Marcus spoke with VentureBeat on Thursday.

“Some of them are somewhat better than GPT-4, but there is no quantum jump. I believe everyone would agree that GPT-4 is a quantum leap forward of GPT-3.5. “There hasn’t been a [quantum leap] in over a year,” Marcus remarked.

As the performance gap narrows and more models produce competitive outcomes, it begs the issue of whether LLMs are approaching a plateau. If this trend continues, it might have serious consequences for the future development and deployment of language models, perhaps moving the emphasis away from merely increasing model size and towards more efficient and specialized architectures.

Drawbacks of the LLM Approach

The LLMs, although strong, have severe downsides. For starters, training LLMs demands a massive quantity of data, with billions or perhaps trillions of parameters. This makes the training process exceedingly resource-intensive, with startling computing and energy requirements for training and running LLMs. This results in hefty expenses, making it harder for smaller organizations or individuals to invest in core LLM development. At an MIT presentation last year, OpenAI CEO Sam Altman claimed that training GPT-4 would cost at least $100 million. 

The complex nature of the tools and approaches needed to deal with LLMs creates a steep learning curve for developers, thus limiting accessibility. Developers have a long cycle time, from training to developing and deploying models, which slows down development and experimentation. A recent report from the University of Cambridge demonstrates that organizations can spend 90 days or more implementing a single machine learning (ML) model.  

Another key difficulty with LLMs is their susceptibility to hallucinations, which result in outputs that appear plausible but are not accurate or factual. This is due to the way LLMs are trained to anticipate the next most probable word based on patterns in training data rather than a genuine grasp of the content. As a result, LLMs may safely make misleading assertions, invent facts, and connect unrelated concepts in illogical ways. Detecting and controlling these hallucinations is a constant issue in the creation of dependable and trustworthy language models.

“If you’re using something for a high-stakes situation, you don’t want to offend your customer, get incorrect medical information, or use it to drive a car and take risks. “That’s still a problem,” Marcus warns.

The size and black-box nature of LLMs can also make them difficult to understand and debug, which is critical for establishing trust in the model’s results. Bias in the training data and algorithms might result in unfair, incorrect, or even destructive results. As demonstrated by Google Gemini, measures used to make LLMs “safe” and dependable can also limit their efficacy. Also, the centralized structure of LLMs raises worries about a few major digital corporations wielding too much power and authority.

Introducing Small Language Models (SLMs)

Enter the small language models. SLMs are more efficient variants of LLMs, with fewer parameters and simpler designs. They need minimal data and training time—minutes or a few hours, as opposed to days with LLMs. This makes SLMs more efficient and simple to set up on-site or on smaller devices.

One of the primary benefits of SLMs is their adaptability for certain applications. Because they have a more narrow scope and need less data, they are easier to fine-tune for certain domains or activities than huge, general-purpose models. This customization allows businesses to build SLMs that are very effective for their unique requirements, such as sentiment analysis, named entity identification, or domain-specific question answering. SLMs’ specialized character might result in better performance and efficiency in certain specific applications than a more generic model.

Another advantage of SLMs is the possibility of increased privacy and security. SLMs are easier to audit and have fewer unexpected vulnerabilities because of their smaller codebase and simpler design. This makes them appealing for applications that handle sensitive data, such as healthcare or banking, where data breaches might result in serious consequences. Also, SLMs have lower processing needs, making them more practical to run locally on devices or on-premises servers rather than relying on cloud infrastructure. This local processing can enhance data security and lower the danger of exposure during data transfer.

In addition, SLMs are less likely than LLMs to experience undetected hallucinations within their specified area. SLMs are often trained on a smaller and more focused dataset particular to their intended domain or application, allowing the model to learn the patterns, language, and information that are most important to its purpose. This concentration decreases the probability of producing irrelevant, unexpected, or inconsistent results. SLMs are less likely to capture and magnify noise or mistakes in training data due to their fewer parameters and more streamlined architecture.

Clem Delangue, CEO of AI firm HuggingFace, estimated that SLMs might solve up to 99% of use cases, and 2024 would be the year of the SLM. HuggingFace, a platform that allows developers to create, train, and deploy machine learning models, announced a strategic agreement with Google earlier this year. HuggingFace has since been incorporated into Google’s Vertex AI, enabling developers to instantly deploy hundreds of models via the Google Vertex Model Garden. 

Show Gemma Some Love, Google

After first losing its lead in LLMs to OpenAI, Google is now aggressively targeting the SLM possibility. In February, Google released Gemma, a new set of tiny language models that are intended to be more efficient and user-friendly. Gemma versions, like other SLMs, may run on a wide range of ordinary devices, including smartphones, tablets, and laptops, without the need for specific hardware or considerable optimization.

Since Gemma’s release last month, the trained models have received over 400,000 downloads on HuggingFace, and a few fascinating projects are already underway. Cerule, for example, is a strong image and language model that blends Gemma 2B with Google’s SigLIP and was trained on a large dataset of pictures and text. Cerule uses very efficient data selection algorithms, implying that it can achieve great performance without requiring a large quantity of data or processing. This suggests that Cerule might be well-suited for upcoming edge computing use cases.  

The Revolutionary Power of Small Language Models

As the AI community continues to investigate the potential of compact language models, the benefits of shorter development cycles, increased efficiency, and the capacity to modify models to specific requirements become more evident. SLMs have the potential to democratize AI access and stimulate innovation across sectors by allowing low-cost, focused solutions. The use of SLMs at the edge offers up new opportunities for real-time, personalized, and secure applications in a variety of industries, including finance, entertainment, automotive systems, education, e-commerce, and healthcare.

Edge computing with SLMs improves user experiences by processing data locally and minimizing dependency on cloud infrastructure. This decentralized AI strategy has the potential to change the way organizations and consumers engage with technology, resulting in more personalized and intuitive experiences in the real world. As LLMs confront computing resource problems and may reach performance plateaus, the advent of SLMs promises to keep the AI ecosystem advancing at a rapid rate.

Source- VentureBeat