draft: update on local models

1 files changed, 97 insertions, 0 deletions

diff --git a/content/posts/an-update-on-local-models.md b/content/posts/an-update-on-local-models.md
new file mode 100644
index 0000000..a916bb5
--- /dev/null
+++ b/content/posts/an-update-on-local-models.md
@@ -0,0 +1,97 @@
+---
+title: ""
+draft: true
+date: 2026-08-22
+lastmod:
+description:
+categories:
+  - LLM
+  - Coding Agents
+---
+
+## Qwen 3.8 27B
+
+This week was interesting for local models, because we had the
+open weights release of Qwen 3.8 27B. It was much expected, because
+Qwen 3.6 held the crown for a long time in open models of a size
+that is runnable on almost consumer hardware.
+
+The [Artificial Analysis](https://artificialanalysis.ai/models/qwen3-8-27b)
+score is pretty damn good! 52 with xhigh is in the ballpark of 
+GPT-5.6 Luna (max) and DeepSeek V4 Flash 0731, which were my go
+to models for august, given their relatively low price.
+To think that I could run as a powerful model locally feels
+insane.
+For reference, it sits between a Claude Opus 4.6 and a
+Claude Opus 4.7, and that was the best what we had in February
+in Claude Code.
+
+## Artificial Analysis pareto frontier
+
+https://artificialanalysis.ai is my reference for model comparison.
+It actually compiles scores from many benchmarks. I know there are
+many pitfalls to these benchmarks, but hey, we have to reduce
+the world's complexity, right?
+I found my prefered charts, and that's when you remove all models.
+Then AAI highlights the pareto frontier, with the best option
+at each tradeoff. Go see for yourself!
+
+
+[Int vs Cost/Time per task](https://artificialanalysis.ai/models#intelligence-comparisons) - What frontier model to use in
+what case. Here we can see Luna is pretty neat for cost, and
+that's what I used in august.
+
+[Int vs Active/Total params](https://artificialanalysis.ai/models#model-size) - This helps me understand what I could
+run on my hardware. For this one, you need to filter on small
+(and maybe tiny) models. This show Qwen 3.8 27B as much better
+than the previous Gemma 4 26B A4B. And I can see a new contender
+[Ling 3.0 Tiny](https://artificialanalysis.ai/models/ling-3-0-tiny)
+that looks promising.
+
+## Running modern models
+
+I wonder if local inference got better since https://blog.kagamino.dev/posts/local-llm-inference-offline-scenario/.
+Back then, I had decent inference performance with Gemma 4 E2B
+(AAI 15), but it felt too dumb to do meaningful work.
+I wanted to test Qwen 3.5 9B, but this time it was too slow,
+on my Desktop as well as on my Mac.
+
+As I ran out of my OpenCode Go subscription for the last ten days
+of august, it's time to check if I can work locally!
+Maybe there were new smaller and smarter models, like Qwen
+or Ling ? Or better inference, like oMLX on Mac?
+
+Last time was a one shot scenario, so this time I want to test
+a coding agentic workflow with Pi - it has a smaller system prompt.
+I'll go by feel, because I don't have clear metrics to target.
+
+Sadly, Qwen 3.8 27B is too large, even with oMLX. That's also
+because I need a larger context of 100k tokens. And I couln't
+run Ling 3.0 Tiny, because of unsupported architecture. So much
+for modern models!
+
+## New contenders
+
+But the AAI pareto frontier gives me new ideas of old models
+to run.
+[Qwen 3.5 4B](https://artificialanalysis.ai/models/qwen3-5-4b)
+is still on this frontier with a AAI 20, and a smaller size
+than the
+[Gemma 4 E2B](https://artificialanalysis.ai/models/gemma-4-e2b/)
+(AAI 10) I tested last time.
+
+With oMLX and a 8bits version, it runs using around 10GB
+unified memory. The first prompt is always a bit slow, but then
+the KV cache kicks in and it feels snappy enough.
+But after some time it ended up in a locked grep loop. I think
+this is usable for targeted changes.
+
+I had an attempt on my desktop, which is better a prefill (input)
+than the Mac. But this time it didn't feel really smart.
+
+In the end, I'm a bit disappointed I couldn't run the newer models,
+even though Ling 3.0 Tiny was supposed to fit. But maybe i can
+run it later when the tooling has caught up.
+I think Qwen 3.5 4B may have a spot for offline use, as I can run
+it at 50 tok/s on Mac. I'd like to investigate using it
+as an autocomplete model for IDE.