Why Every Business Should Understand Local AI

For the past six months I’ve been consistently experimenting with local AI models. Pretty much every week night.

Not because I wanted to replace ChatGPT, Claude or Gemini. Quite the opposite.

It started with curiosity, more a compulsion.

Could an independent consultant, without a research lab or a huge budget, build something genuinely useful with local AI?

The answer turned out to be yes.

But It hasn’t been glamorous. Still isn’t.

I started on my MacBook Air with 8 GB RAM barely enough memory to cope, running Phi-3 through the terminal. The hallucinations were hilarious, sometimes sounding as though they’d come straight out of the 1950s. After that machine finally gave up on a death by restart, I found myself running Gemma 2B on a 15-year-old Intel Mac, watching it generate roughly one word per second. (watching it generate roughly one word per second. It reminded me of sitting in an internet café in Havana in 2004, waiting what felt like half an hour just to send a single email.).

Since then I’ve experimented with different hardware and models, gradually learning what works and, more importantly, what doesn’t. Along the way I also answered a question that had been bothering me from the start:

How can someone without a research lab build a local AI workspace around their own knowledge? Not to compete with frontier models, but to understand what it feels like when the knowledge belongs to you rather than the platform.

The biggest lesson wasn’t about the model itself.

It was about ownership.

At some point I realised that running a local model isn’t just about asking questions offline.

It’s about creating a workspace.

A place where your own documents, frameworks and knowledge can live alongside the model, allowing it to retrieve and work with information that’s specific to you all the while you maintain control over your information. The model has its own library.

The work not only focuses on the model itself but also towards the quality of the knowledge I give it. Every useful AI needs context. Mine just happens to come from my own documents, notes and frameworks.

That was a penny drop for me.

The recent discussions around temporary access restrictions to frontier models reminded me why this is important. We all got the jitters when the US government cut the tap on Claude Fable.

But I’m not arguing that everyone should abandon cloud AI.

For most people, services like ChatGPT and Claude remain extraordinary tools, with abundant access, for now. That won’t continue.

So I do think there’s value in understanding what local AI can offer, if only as part of a broader strategy for resilience and independence.

I’m still very much learning, and that’s part of the appeal.

Michael