Jose Romero
← All posts
Prefer email? Read this on Substack

Local AI Is a Scam? My Honest Take, and What I Actually Use It For

According to my comment section, local models are overrated, a huge scam and a dumb, expensive hobby. Because of that hobby, AI is causing RAM shortages, hardware shortages, gamers are suffering, and so on. To be fair, there's some truth to all of that. Local AI is not for everyone and it does not replace a frontier lab subscription. So this one is a rant: the good, the bad, my honest take, and what I actually use my Strix Halo box for.

It's not the rig of my dreams. It's not a bunch of RTX 6000s and a stack of DGX Sparks. It's a small box, and I do a lot with it.

The opening of the video: Jose with comment stickers saying local AI is overrated, a huge scam and a dumb, expensive hobby

The good

Compared to two years ago, local AI is in a really good place. The models are shockingly good now. They're a lot smaller than they used to be and much smarter for their size. Open licenses mean people take them, tweak them and make them better.

The harnesses around the models got better too: Hermes Agent (I can never figure out how to pronounce it), Pi, OpenCode, Claude Code and OpenAI's Codex. Take a decently smart model on reasonable hardware, add a really good harness, MCPs and tools, and you can get a lot done with something some people would call not enough.

Hardware caught up as well. Apple runs models very well, there's Strix Halo, the new Gorgon Halo chips, and Nvidia's GB10 boxes, which are sold out and stupid expensive. Hardware is no longer the bottleneck. It's a price problem now. And the tools are easier than ever: read a few guides, watch a few videos, and in five or ten minutes you have a local model running, even on a phone.

Hardware is no longer the problem: 'hardware problem' struck through and 'a price problem' beside Jose

The bad

There are a lot fewer guardrails. You don't get the cushions of a ChatGPT or Claude subscription, where they really baby you. Dive into gray territory like medical questions or security work and a cloud model shuts you off quickly. Local models mostly don't, and that's probably intentional, because nobody wants to neuter something that's meant to be free and open.

Cloud AI shuts you off fast on medical, security and red team questions

There's still a ton of tinkering: the harness, the model, your hardware, maybe running without a display. You end up learning what MTP and n-grams are, terms you may not even want to know. I like the back end stuff. Most people won't.

Uncensored models have even fewer guardrails. Sometimes they're dumber than the originals, but they won't tell you no, and that puts a bad taste in regulators' mouths. And now there are poisoned models: give one a code phrase and it can start sending your environment variables to a server. I made a whole video about that.

My honest take

For most people, cloud frontier models are the easiest and biggest return on investment. I'm never going to set someone who has never touched AI up with a local model. You sign up for a $20 plan and you're chatting with a bot within minutes. The latest models get much better results with less pain and friction. Local models are always behind, and to get comparable results you have to babysit them, use a good harness, tinker and add guardrails.

If you buy hardware expecting to get your money back, you most likely won't. My guess is two to five years to replace a subscription, and that assumes the box runs almost nonstop, before electricity, maintenance and your own time. The exception is if you sell it as a service to people who care about security and compliance more than the latest model.

Money back? Probably not: two to five years running almost nonstop, before electricity, maintenance and your time

Local AI is also more than a web UI that copies ChatGPT. There's a whole world of specialized models to connect together, and it changes daily. I took a couple of weeks off and came back to three new model releases.

What I actually use it for

Mostly privacy, max privacy. The box sits at home on my network, and I reach it over Tailscale.

A declutter bot. I forward Hermes pictures, ideas, screenshots and plans. It runs on my local box and files them into Obsidian: a place to visit or eat, a quote, a date night idea. No more digging through screenshots from months ago.

The declutter bot: I forward it anything, Hermes on my box files it in Obsidian

My own document library. PDFs, docs, Excel sheets, Markdown and HTML files, school records, medical files, bills, contracts. Local AI extracts the contents, writes the scripts, does the transcriptions and checks the result with vision. I was basically doing retrieval-augmented generation (RAG) before I knew the name. A good Qwen model, a good harness and context about me, on a modest box, can do a lot, and none of it sits in someone's cloud drive.

RAG, retrieval-augmented generation: a good Qwen model, a good harness and context about me

A life retro. I've journaled for years, on paper and digitally, and I record voice memos. Small transcription models turn those into text, and then I ask things I'd never send to Anthropic or OpenAI: what did I do on this date five years in a row, what was I thinking about this five years ago, what were my goals this year. It's a little cheesy, but I use it for gratitude too.

Questions for my own AI: what did I do on this date five years in a row, what was I thinking five years ago, what were my goals this year

Finances. CSV files and bank statements get organized and categorized into charts and a debt payoff plan. I started from a YNAB-style budget structure and had AI adapt it to our finances.

Health. Apple Health exports, a smart scale, blood work, doctor's notes and supplements go into one folder for a small model that specializes in medicine. It's not my doctor. It explains trends, and I take the questions to my doctor, who usually agrees.

Not groundbreaking, just private: declutter bot, document library, life retro, finances, health

None of this is groundbreaking. It just makes my life easier in a more private way.

Where I think it's going

Who knows what happens when the AI market pops, subscription prices go through the roof and tokens get metered harder than ever. Local AI might become the only option, or a luxury. My bet is that local AI is the future: smaller models tailored to specific tasks on modest hardware. Massive data centers and subscriptions running at a loss don't look sustainable to me.

So if it's privacy focused, go local. For anything else, look at the cheap subscriptions. I have both: the local box, a Claude subscription and a $20 OpenAI plan. Behind the scenes, Claude Code designs my benchmarks at max settings, local Qwen 3.8 Flash-Next runs them for days for free, and GPT-6 Astra checks Qwen's work, Claude's work and my assumptions.

How I run my benchmarks: Claude Code designs, local models run it for free, Astra checks it all

What are your top three local AI use cases? Tell me what I'm missing.

Sources

🎥 Gear links

Affiliate links may earn me a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.

Get new posts

Every post goes out free on Substack. What I build, break, and fix, written up in plain English. No spam, unsubscribe anytime.

Subscribe on Substack