As businesses progressively incorporate artificial intelligence (AI) into their operations, the cost and complexity of working with large language models (LLMs) are coming into focus.
Generative AI is transforming business operations by enabling automation, faster decision-making, and enhanced customer engagement. But using LLMs often requires enormous computing power, expensive cloud infrastructure, and vast data storage. It also raises concerns about data privacy on public platforms. This can put LLMs out of reach for smaller enterprises and public organizations.
Small language models (SLMs) like Microsoft’s Phi-3 model offer an alternative that can extend AI systems beyond large enterprises with deep pockets to small businesses, startups, and even schools.
How can SLMs provide a solution to make AI viable for these organizations?
- Small language models like Microsoft’s Phi-3 model provide cost-effective, efficient AI that can run offline on local devices.
- Unlike large language models, SLMs reduce reliance on expensive cloud infrastructure and massive computing power.
- Offline AI chatbots and assistants powered by SLMs offer practical solutions for small businesses, schools, and healthcare providers.
- Processing data locally improves privacy, compliance, and trust by ensuring sensitive information never leaves the device.
- While SLMs are less powerful than LLMs, they balance accessibility, affordability and usability for everyday tasks.
What Are Small Language Models?
SLMs are lightweight versions of AI models that typically contain fewer than 10 billion parameters, referring to how many complex instructions models can understand – compared with around 400 billion in LLMs. These smaller models can run directly on local hardware, such as laptops, desktops, or edge devices.
LLMs are typically trained on large, diverse datasets that incorporate a broad range of texts, including books, articles, and websites, so that they can learn and interpret the structural patterns and nuances of a language and use this to generate intelligent responses to virtually any prompt. SLMs are trained on smaller, focused datasets and deliver accuracy comparable to LLMs. They are designed to be fine-tuned for specific, task-focused applications.
This allows them to deliver targeted capabilities – such as chatbots, learning aids, and writing assistants – that can run offline, so that organizations with limited budgets or sensitive information can leverage AI without the high costs, cloud dependency, and risk of data privacy breaches that come with large-scale systems.
SLMs are optimized for speed and efficiency. They are well-suited to real-time applications, edge devices, and environments with limited internet access or strict data privacy requirements.
Small language model examples include:
Microsoft Phi-3 Model: A Game Changer for Offline AI
Microsoft’s open-source Phi-3 family is considered to be one of the best SLMs available, as it can outperform models of similar and even larger sizes on benchmarks including language, reasoning, coding, and math.
It is trained on a smaller dataset than LLMs like GPT-4. Released in 2024, the Microsoft Phi-3 model includes Phi-3-mini (3.8 billion parameters), Phi-3-small (7B), and Phi-3-medium (14B).
ONNX Runtime optimization with support for Windows DirectML allows Phi-3-mini to run locally on laptops and mobile devices.
The smaller size makes Phi-3 easier and more affordable to fine-tune or customize. The model provides 128,000 tokens of context length, which enables it to handle long documents or tasks that involve deep reasoning without losing context.
Microsoft says that its experience in delivering copilots and working with businesses to deploy generative AI through its Azure AI platform highlighted a growing need for different size models for different tasks.
It developed Phi-3 for resource-constrained environments, such as on devices and offline inferencing, applications where fast response times are critical, and simpler low-cost tasks.
Like other AI models, Phi-3 has learned from previous iterations to become more sophisticated. Phi-1 was focused on coding, while Phi-2 began to learn reasoning, and Phi-3 has improved coding and reasoning capabilities.
Smaller models like Phi-3 can work better for custom applications, as organizations tend to have small, specific internal datasets that they want to train models on and do not need the full breadth of an LLM.
Small businesses and schools can use Phi-3 to:
- Deploy offline AI tools such as customer support chatbots or internal assistants securely on-premises.
- Automate routine tasks such as drafting emails or generating reports.
- Generate content such as product descriptions, marketing copy, or social media posts without cloud tools.
- Provide educational tools in schools, such as offline AI tutoring assistants without needing constant internet access.
- Reduce operational costs thanks to lower compute requirements.
Competitors to Phi-3, like Google’s Gemma 2B and 7B, target simpler tasks like document summarization or coding assistance, and Llama 3 8B can also be used for some chatbots. Anthropic’s Claude 3 Haiku can quickly summarize dense research papers.
Why SLMs Work for Enterprise Applications
Offline AI chatbots based on SLMs like the Phi-3 model are particularly well-suited to small enterprises and schools that face budget constraints or strict data privacy regulations.
With SLM-based systems, users do not need to pay ongoing cloud subscription fees. SLMs can maximize hardware use, reducing power consumption and heat generation, in turn, lowering operational costs. The data is processed locally and never leaves the device, which is key for organizations that are required to meet strict privacy regulations such as GDPR or HIPAA.
Cloud-based systems can be vulnerable to hacking, but there is a lower risk of breaches in offline AI systems, as the attack surface is smaller since the information never leaves the local environment. Parents and employees can have more trust in the safety of these systems as they know their personal data is not being shared with external platforms or third-party providers.
As SLMs are purpose-built for specific applications, they can be fine-tuned for greater precision while maintaining efficiency.
And even in areas with poor internet access, SLMs running offline can help ensure critical services like record-keeping, tutoring, or diagnostics continue without disruption.
Limitations of Phi-3 & Other SLMs
While Phi-3 and other small language models are a step forward in enabling broader AI adoption, it is important to be aware of their limitations:
- Smaller training datasets can reduce overall knowledge.
- Lower accuracy than large-scale LLMs on complex reasoning tasks.
- Limited customization, although fine-tuning is possible.
- Like all models, SLMs may reflect societal biases in training data.
- May have a limited range of programming languages
SLMs will attain some general knowledge from their training data, but they are unlikely to rival a large-scale LLM in breadth. The quality of answers from an LLM trained on the internet will naturally differ from a smaller model like Phi-3.
Overall, SLMs are well-suited to certain everyday tasks, but not yet ready to challenge LLMs in advanced applications.
Before investing in an AI model, organizations should consider whether an SLM vs. LLM would be a more appropriate fit for their use cases and requirements. It is important to start by defining the purpose of the system they want to introduce. This can help determine whether AI adoption makes sense.
Considerations include:
- Use case complexity – Does the task require advanced reasoning and broad knowledge (LLM) or lighter, task-focused capabilities (SLM)?
- Infrastructure requirements – Can existing hardware support the model, or will it need cloud services?
- Budget – Can the organization afford the costs of cloud infrastructure and subscriptions, or would a lightweight offline AI model be more sustainable?
- Data privacy and compliance – Is sensitive information involved that should remain on-device rather than being processed in the cloud?
- Scalability – Is the goal to support a small, focused deployment, or scale across large teams and applications? Are there resources for scaling as usage grows?
- Connectivity needs – Will the system need to function offline, or will it have uninterrupted internet access?
- Performance – Is speed and efficiency more important than high accuracy and coverage?
- Customization potential – Will the model need fine-tuning for a specific industry or audience?
The future of enterprise AI is about scaling intelligently – implementing AI solutions that are appropriate to the requirements of the specific use case to ensure efficiency.
The rise of small language models like Microsoft’s Phi-3 is making AI more accessible. For small businesses and schools, these offline AI solutions strike a balance between cost, privacy, and usability.
While they are unable to match the full power of massive cloud-based models, they represent a practical, scalable way for smaller organizations to incorporate AI on their own terms.
Is there an AI that works offline?
Yes. Microsoft’s Phi-3 model and other small language models (SLMs) can run locally on devices without an internet connection.
Can AI be run offline?
Yes. SLMs are designed for offline deployment on consumer-level devices. With SLMs like Phi-3, businesses can deploy offline AI chatbots, content generators, and assistants on local hardware.
What are the limitations of Phi-3?
The Phi-3 model is lightweight and efficient, but the tradeoff is potentially lower accuracy, reduced knowledge coverage, and fewer customization options compared to larger models.