Small AI models locally hosted might just be one of the most underrated moves in applied AI right now. In this episode, I share why I’m exploring locally hosted smaller models as part of building Voice Studio, and why small can genuinely be powerful when the job is well defined.
Why Small AI Models Locally Can Be a Serious Choice
The public conversation about AI is dominated by scale — bigger, faster, more capable. But building Voice Studio has pushed me in a very different direction, and I want to share what I’m learning.
Voice Studio is being developed as a local AI system using Qwen, running on a local server rather than depending on a public cloud chatbot, retrieving information from my own structured knowledge base. And the more I build it, the more I believe that small, focused and local is a genuinely powerful approach — not for every task, but for the right ones.
Three Practical Reasons I’m Going Local
1. Ownership and Control
When a system works with confidential client material and my own intellectual property, I need to understand where that information sits, how it’s retrieved and which services can access it. Local hosting doesn’t automatically solve every privacy or governance issue — you still need good architecture, access controls and monitoring — but it gives you more control over those decisions. That matters.
2. Matching the Model to the Work
Not every task in my business needs to go to the largest frontier model available. When a task is narrow, the knowledge is structured and the outcome is clear, a smaller model can be more efficient and more economical. Combine that with retrieval-augmented generation pulling in current, trusted source material, and you have a system that’s genuinely useful for a defined domain.
3. Cost Structure
Frontier models are extraordinary — I use them every day. But repeated use at scale has a real price. A locally hosted model requires infrastructure and technical management, so it’s not free either, but it creates a different cost structure for frequent, focused tasks. When you’re designing something meant to become part of everyday operations, that difference matters.
The European Dimension Worth Talking About
There’s a broader question here that I think public sector professionals especially need to sit with. Which models do we rely on? Where are they developed? Under what licences? Where is the infrastructure located? These aren’t abstract geopolitical questions — they directly influence procurement, data strategy, compliance and the ability to build systems in the public interest.
And a quick note on language: people often use “open source” as a blanket term for AI models, but many are more accurately described as open-weight. The model weights may be available, while the training data or commercial terms may not be open in the same way. If ownership and control are among your reasons for choosing a model, those distinctions need to be understood rather than assumed.
More Control Means More Responsibility
Building locally isn’t a shortcut. Someone has to maintain the infrastructure, update the model, test performance, manage access and monitor failures. You gain control, but you also accept responsibility for decisions a cloud platform would normally handle on your behalf.
What excites me is the pathway Voice Studio has opened up. If I can build one locally hosted system around my own knowledge, I can build others around defined tasks and domains — smaller, more purposeful, closer to the organisation they serve.
The next phase of applied AI won’t belong only to the companies building the world’s largest models. It will also belong to organisations that learn how to combine the right model, trusted knowledge and a clearly defined purpose.
If you want to build your AI literacy and skills, I can help
📚 Grab AI Playbook on Amazon, the perfect podcast companion.
📰 Subscribe to Human in the Loop, my LinkedIn Newsletter
🎓 Take the CPD Accredited AI Playbook Diploma
📞 Book a call at digitaltraining.ie
About the Podcast Host
AI and Communications Specialist | Author | Podcaster | International Speaker
Joanne Sweeney is a recognised leader in digital marketing and digital transformation, specialising in the practical application of AI to enhance communication strategies. The founder of Digital Training Institute, Joanne, empowers leaders and communications professionals to leverage AI responsibly and effectively, ensuring transparency, trust, and tangible results.
She completed the Harvard Kennedy School Executive Programme in AI and Public Policy this summer before launching her fourth book AI Playbook, in September. Joanne has also published two editions of Public Sector Marketing Pro, which were preceded by her first title, Social Media Under Investigation Law Enforcement and the Social Web.
Her books have become go-to resources for governments, public sector and policing agencies, as well as marketing and communications professionals, all navigating the challenges of the changing Internet.
Joanne is the host of the AI SIX Podcast, a daily six-minute podcast that provides actionable insights, expert interviews, and practical advice to help professionals navigate the new world of work.
