Iain Harper's Blog

Caveat: this article contains a detailed examination of the state of open source/ weight AI technology that is accurate as of February 2026. Things move fast.

I don’t make a habit of writing about wonky AI takes on social media, for obvious reasons. However, a post from an AI startup founder (there are seemingly one or two out there at the moment) caught my attention.

His complaint was that he was spending $1,000 a week on API calls for his AI agents, realised the real bottleneck was infrastructure rather than intelligence, and dropped $10,000 on a Mac Studio with an M3 Ultra and 512GB of unified memory. His argument was essentially every model is smart enough, the ceiling is infrastructure, and the future belongs to whoever removes the constraints first.

It’s a beguiling pitch and it hit a nerve because the underlying frustration is accurate. Rate limits, per-token costs, and context window restrictions do shape how people build with these models, and the desire to break free of those constraints is understandable. But the argument collapses once you look at what local models can actually do today compared to what frontier APIs deliver, and why the gap between the two is likely to persist for the foreseeable future.

To understand why, you need to look at the current open-source model ecosystem in some detail, examine what’s actually happening on the frontier, and think carefully about the conditions that would need to hold for convergence to happen.

The open-source ecosystem in early 2026

The open-source model ecosystem has matured considerably over the past eighteen months, to the point where dismissing it as a toy would be genuinely unfair. The major families that matter right now are Meta’s Llama series, Alibaba’s Qwen line, and DeepSeek’s V3 and R1 models, with Mistral, Google’s Gemma, and Microsoft’s Phi occupying important niches for specific use cases.

DeepSeek’s R1 release in January 2025 was probably the single most consequential open-source event in the past two years. Built on a Mixture of Experts architecture with 671 billion total parameters but only 37 billion activated per forward pass, R1 achieved performance comparable to OpenAI’s o1 on reasoning benchmarks including GPQA, AIME, and Codeforces. What made it seismic was the claimed training cost: approximately $5.6 million, compared to the hundred-million-dollar-plus budgets associated with frontier models from the major Western labs. NVIDIA lost roughly $600 billion in market capitalisation in a single day when the implications sank in.

The Lawfare Institute’s analysis of DeepSeek’s achievement noted an important caveat that often gets lost in the retelling: the $5.6 million figure represents marginal training cost for the final R1 phase, and does not account for DeepSeek’s prior investment in the V3 base model, their GPU purchases (which some estimates put at 50,000 H100-class chips), or the human capital expended across years of development. The true all-in cost was substantially higher. But even with those qualifications, the efficiency gains were highly impressive, and they forced the entire industry to take algorithmic innovation as seriously as raw compute scaling.

Alibaba’s Qwen3 family, released in April 2025, pushed things further. The 235B-A22B variant uses a similar MoE approach, activating 22 billion parameters out of 235 billion, and it introduced hybrid reasoning modes that can switch between extended chain-of-thought and direct response depending on task complexity. The newer Qwen3-Coder-480B-A35B, released later in 2025, achieves 61.8% on the Aider Polyglot benchmark under full precision, which puts it in the same neighbourhood as Claude Sonnet 4 and GPT-4.1 for code generation specifically.

Meta’s Llama 4, released in early 2025, moved to natively multimodal MoE with the Scout and Maverick variants processing vision, video, and text in the same forward pass. Mistral continued to punch above its weight with the Large 3 release at 675 billion parameters, and their claim of delivering 92% of GPT-5.2’s performance at roughly 15% of the price represents the kind of value proposition that makes enterprise buyers think twice about their API contracts.

According to Menlo Ventures’ mid-2025 survey of over 150 technical leaders, open-source models now account for approximately 13% of production AI workloads, with the market increasingly structured around a durable equilibrium. Proprietary systems define the upper bound of reliability and performance for regulated or enterprise workloads, while open-source models offer cost efficiency, transparency, and customisation for specific use cases.

By any measure, this is a serious and capable ecosystem. The question is whether it’s capable enough to replace frontier APIs for agentic, high-reasoning work.

What happens when you run these models locally

The Mac Studio with an M3 Ultra and 512GB of unified memory is genuinely impressive hardware for local inference. Apple’s unified memory architecture means the GPU, CPU, and Neural Engine all share the same memory pool without the traditional separation between system RAM and VRAM, which makes it uniquely suited to running large models that would otherwise require expensive multi-GPU setups. Real-world benchmarks show the M3 Ultra achieving approximately 2,320 tokens per second on a Qwen3-30B 4-bit model, which is competitive with an NVIDIA RTX 3090 while consuming a fraction of the power.

But the performance picture changes dramatically as model size increases. Running the larger Qwen3-235B-A22B at Q5 quantisation on the M3 Ultra yields generation speeds of approximately 5.2 tokens per second, with first-token latency of around 3.8 seconds. At Q4KM quantisation, users on the MacRumors forums report around 30 tokens per second, which is usable for interactive work but a long way from the responsiveness of cloud APIs processing multiple parallel requests on clusters of H100s or B200s. And those numbers are for the quantised versions, which brings us to the core technical problem.

Quantisation is the process of reducing the numerical precision of a model’s weights, typically from 16-bit floating point down to 8-bit or 4-bit integers, in order to shrink the model enough to fit in available memory. The trade-off is information loss, and research published at EMNLP 2025 by Mekala et al. makes the extent of that loss uncomfortably clear. Their systematic evaluation across five quantisation methods and five models found that while 8-bit quantisation preserved accuracy with only about a 0.8% drop, 4-bit methods led to substantial losses, with performance degradation of up to 59% on tasks involving long-context inputs. The degradation worsened for non-English languages and varied dramatically between models and tasks, with Llama-3.1 70B experiencing a 32% performance drop on BNB-nf4 quantisation while Qwen-2.5 72B remained relatively robust under the same conditions.

Separate research from ACL 2025 introduces an even more concerning finding for the long-term trajectory of local models. As models become better trained on more data, they actually become more sensitive to quantisation degradation. The study’s scaling laws predict that quantisation-induced degradation will worsen as training datasets grow toward 100 trillion tokens, a milestone likely to be reached within the next few years. In practical terms, this means that the models most worth running locally are precisely the ones that lose the most from being compressed to fit.

When someone says they’re using a local model, they’re usually running a quantised version of an already-smaller model than the frontier labs deploy. The experience might feel good in interactive use, but the gap becomes apparent on exactly the tasks that matter most for production agentic work. Multi-step reasoning over long contexts, complex tool use orchestration, and domain-specific accuracy where “pretty good” is materially different from “correct.”

The post-training gap that open source can’t easily close

The most persistent advantage that frontier models hold over open-source alternatives has less to do with architecture and more to do with what happens after pre-training. Reinforcement Learning from Human Feedback and its variants form a substantial part of this gap, and the economics of closing it are unfavourable for the open-source community.

RLHF works by having human annotators evaluate pairs of model outputs and indicate which response better satisfies criteria like helpfulness, accuracy, and safety. Those preferences train a reward model, which then guides further optimisation of the language model through reinforcement learning. The process turns a base model that just predicts the next token into something that follows instructions well, pushes back when appropriate, handles edge cases gracefully, and avoids the confident-but-wrong failure mode that plagues undertrained systems.

The cost of doing this well at scale is staggering. Research from Daniel Kang at Stanford estimates that high-quality human data annotation now exceeds compute costs by up to 28 times for frontier models, with the data labelling market growing at a factor of 88 between 2023 and 2024 while compute costs increased by only 1.3 times. Producing just 600 high-quality RLHF annotations can cost approximately $60,000, which is roughly 167 times more than the compute expense for the same training iteration. Meta’s post-training alignment for Llama 3.1 alone required more than $50 million and approximately 200 people.

The frontier labs have also increasingly moved beyond basic RLHF toward more sophisticated approaches. Anthropic’s Constitutional AI has the model critique its own outputs against principles derived from human values, while the broader shift toward expert annotation, particularly for code, legal reasoning, and scientific analysis, means the humans providing feedback need to be domain practitioners rather than general-purpose annotators. This is expensive, slow, and extremely difficult to replicate through the synthetic and distilled preference data that open-source projects typically rely on.

The 2025 introduction of RLTHF (Targeted Human Feedback) from research surveyed in Preprints.org offers some hope, achieving full-human-annotation-level alignment with only 6-7% of the human annotation effort by combining LLM-based initial alignment with selective human corrections. But even these efficiency gains don’t close the fundamental gap: frontier labs can afford to spend tens of millions on annotation because they recoup it through API revenue, while open-source projects face a collective action problem where the cost of annotation is concentrated but the benefits are distributed.

Where the gap genuinely is closing

The picture is not uniformly bleak for open-source, and understanding where the gap has closed is as important as understanding where it hasn’t.

Code generation is the domain where convergence has happened fastest. Qwen3-Coder’s 61.8% on Aider Polyglot at full precision puts it within striking distance of frontier coding models, and the Unsloth project’s dynamic quantisation of the same model achieves 60.9% at a quarter of the memory footprint, which represents remarkably small degradation. For writing, editing, and iterating on code, a well-configured local model running on capable hardware is now a genuinely viable alternative to an API, provided you’re not relying on long-context reasoning across an entire codebase.

Classification, summarisation, and embedding tasks have been viable on local models for some time, and the performance gap for these workloads is now negligible for most practical purposes. Document processing, data extraction, and content drafting all fall into the category where open-source models deliver sufficient quality at dramatically lower cost.

The OpenRouter State of AI report’s analysis of over 100 trillion tokens of real-world usage data shows that Chinese open-source models, particularly from Alibaba and DeepSeek, have captured approximately 13% of weekly token volume with strong growth in the second half of 2025, driven by competitive quality combined with rapid iteration and dense release cycles. This adoption is concentrated in exactly the workloads described above: high-volume, well-defined tasks where cost efficiency matters more than peak reasoning capability.

Privacy-sensitive applications represent another area where local models have an intrinsic advantage that no amount of frontier improvement can overcome. MacStories’ Federico Viticci noted that running vision-language models locally on a Mac Studio for OCR and document analysis bypasses the image compression problems that plague cloud-hosted models, while keeping sensitive documents entirely on-device. For regulated industries where data sovereignty matters, local inference is a feature that frontier APIs cannot match.

What convergence would actually require

If the question is whether open-source models running on consumer hardware will eventually match frontier models across all tasks, the honest answer requires examining several conditions that would need to hold simultaneously.

The first is that Mixture of Experts architectures and similar efficiency innovations would need to continue improving at their current rate, allowing models with hundreds of billions of total parameters to activate only the relevant subset for each task while maintaining quality. The early evidence from DeepSeek’s MoE approach and Qwen3’s hybrid reasoning is encouraging, but there appear to be theoretical limits to how sparse activation can get before coherence suffers on complex multi-step problems.

The second condition is that the quantisation problem would need a genuine breakthrough rather than incremental improvement. The ACL 2025 finding that better-trained models are more sensitive to quantisation is a structural headwind that current techniques are not on track to solve. Red Hat’s evaluation of over 500,000 quantised model runs found that larger models at 8-bit quantisation show negligible degradation, but the story at 4-bit, where you need to be for consumer hardware, is considerably less encouraging for anything beyond straightforward tasks.

The third and most fundamental condition is that the post-training gap would need to close, which requires either a dramatic reduction in the cost of expert human annotation or a breakthrough in synthetic preference data that produces equivalent alignment quality. The emergence of techniques like RLTHF and Online Iterative RLHF suggests the field is working on this, but the frontier labs are investing in these same efficiency gains while simultaneously scaling their annotation budgets. It’s a race where both sides are accelerating, and the side with revenue-funded annotation budgets has a structural advantage.

The fourth condition is that inference hardware would need to improve enough to make unquantised or lightly quantised large models viable on consumer devices. Apple’s unified memory architecture is the most promising path here, and the progression from M1 to M4 chips has been impressive, but even the top-spec M3 Ultra at 512GB can only run the largest MoE models at aggressive quantisation levels. The next generation of Apple Silicon with 1TB+ unified memory would change the calculus significantly, but that’s likely several years away, and memory costs just shot through the ceiling.

Given all of these dependencies, a realistic timeline for broad convergence across most production tasks is probably three to five years, with coding and structured data tasks converging first, creative and analytical tasks following, and complex multi-step reasoning with tool use remaining a frontier advantage for the longest.

The hybrid approach and what it means in practice

The most pragmatic position right now (which is also the least satisfying one to post about), is that the future is hybrid rather than either-or. The smart deployment pattern routes high-volume, lower-stakes tasks to local models where the cost savings compound quickly and the quality gap is negligible, while reserving frontier API calls for the work that demands peak reasoning: complex multi-step planning, high-stakes domain-specific analysis, nuanced tool orchestration, and anything where being confidently wrong carries real cost.

This is approximately what the Menlo Ventures survey data suggests enterprise buyers are doing already, with model API spending more than doubling to $8.4 billion while open-source adoption stabilises around 13% of production workloads. The enterprises that are getting value from local models are not using them as wholesale API replacements; they’re using them as a complementary layer that handles the grunt work while the expensive models handle the hard problems.

There’s also the operational burden that is rarely mentioned in relation to model use. When you run models locally, you effectively become your own ML ops team. Model updates, quantisation format compatibility, prompt template differences across architectures, memory management under load, and testing when new versions drop, all of that falls on you. The API providers handle model improvements, scaling, and infrastructure, and you get a better model every few months without changing a line of code. For a small team that should be spending its time on product rather than infrastructure, that operational overhead has real cost even if it doesn’t show up on an invoice.

The future of AI probably does involve substantially more local compute than we have today. Costs will come down, architectures will improve, hardware will get more capable, and the hybrid model will become standard practice. The question is not who removes the constraints first, it’s who understands which constraints actually matter.

I am a partner in Better than Good. We help companies make sense of technology and build lasting improvements to their operations. Talk to us today: https://betterthangood.xyz/#contact