diff options
| author | Martin Kagamino Lehoux <martin@lehoux.net> | 2026-08-26 11:40:59 +0200 |
|---|---|---|
| committer | Martin Kagamino Lehoux <martin@lehoux.net> | 2026-08-26 11:40:59 +0200 |
| commit | b6a92239c297248b7072e8b31d4a0174f9213d85 (patch) | |
| tree | fddb91773f1aa2d3173dc13754b0b591780d10d0 | |
| parent | 976a43118d79fd930a2f0500b094c3cee2552399 (diff) | |
new article
| -rw-r--r-- | README.md | 1 | ||||
| -rw-r--r-- | content/posts/new-tests-around-local-llms.md (renamed from content/posts/an-update-on-local-models.md) | 49 |
2 files changed, 27 insertions, 23 deletions
@@ -6,6 +6,7 @@ - [ ] Table of content: `tableOfContents: true`: Works, but too big - [ ] Multi parts series +- [ ] date route structure? ## Guidelines diff --git a/content/posts/an-update-on-local-models.md b/content/posts/new-tests-around-local-llms.md index a916bb5..77e446c 100644 --- a/content/posts/an-update-on-local-models.md +++ b/content/posts/new-tests-around-local-llms.md @@ -1,9 +1,12 @@ --- -title: "" -draft: true -date: 2026-08-22 -lastmod: -description: +title: "New tests around local LLMs" +draft: false +date: 2026-08-26 +lastmod: 2026-08-26 +description: | + Qwen 3.8 27B is an exciting release, but still too large + for common dev machines. But I can find interesting models + to run locally. categories: - LLM - Coding Agents @@ -11,8 +14,7 @@ categories: ## Qwen 3.8 27B -This week was interesting for local models, because we had the -open weights release of Qwen 3.8 27B. It was much expected, because +This week was interesting for local models with the open-weights release of Qwen 3.8 27B. It was highly anticipated, because Qwen 3.6 held the crown for a long time in open models of a size that is runnable on almost consumer hardware. @@ -20,37 +22,38 @@ The [Artificial Analysis](https://artificialanalysis.ai/models/qwen3-8-27b) score is pretty damn good! 52 with xhigh is in the ballpark of GPT-5.6 Luna (max) and DeepSeek V4 Flash 0731, which were my go to models for august, given their relatively low price. -To think that I could run as a powerful model locally feels +To think that I could run as powerful a model locally feels insane. For reference, it sits between a Claude Opus 4.6 and a -Claude Opus 4.7, and that was the best what we had in February +Claude Opus 4.7, and that was the best available in February in Claude Code. ## Artificial Analysis pareto frontier -https://artificialanalysis.ai is my reference for model comparison. +[Artificial Analysis](https://artificialanalysis.ai) is my reference for model comparison. It actually compiles scores from many benchmarks. I know there are -many pitfalls to these benchmarks, but hey, we have to reduce +many pitfalls to these benchmarks, but hey, I have to reduce the world's complexity, right? -I found my prefered charts, and that's when you remove all models. +I found my preferred charts when removing all models. Then AAI highlights the pareto frontier, with the best option -at each tradeoff. Go see for yourself! +at each tradeoff. [Int vs Cost/Time per task](https://artificialanalysis.ai/models#intelligence-comparisons) - What frontier model to use in -what case. Here we can see Luna is pretty neat for cost, and +what case. Here Luna looks pretty neat for cost, and that's what I used in august. [Int vs Active/Total params](https://artificialanalysis.ai/models#model-size) - This helps me understand what I could -run on my hardware. For this one, you need to filter on small -(and maybe tiny) models. This show Qwen 3.8 27B as much better +run on my hardware. For this one, filtering on small +(and maybe tiny) models helps. This shows Qwen 3.8 27B as much better than the previous Gemma 4 26B A4B. And I can see a new contender [Ling 3.0 Tiny](https://artificialanalysis.ai/models/ling-3-0-tiny) that looks promising. ## Running modern models -I wonder if local inference got better since https://blog.kagamino.dev/posts/local-llm-inference-offline-scenario/. +I wonder if local inference got better since my +[previous attempt](/posts/local-llm-inference-offline-scenario/). Back then, I had decent inference performance with Gemma 4 E2B (AAI 15), but it felt too dumb to do meaningful work. I wanted to test Qwen 3.5 9B, but this time it was too slow, @@ -63,12 +66,12 @@ or Ling ? Or better inference, like oMLX on Mac? Last time was a one shot scenario, so this time I want to test a coding agentic workflow with Pi - it has a smaller system prompt. -I'll go by feel, because I don't have clear metrics to target. +I go by feel, because I don't have clear metrics to target. Sadly, Qwen 3.8 27B is too large, even with oMLX. That's also -because I need a larger context of 100k tokens. And I couln't -run Ling 3.0 Tiny, because of unsupported architecture. So much -for modern models! +because I need a larger context of 100k tokens. And I couldn't +run Ling 3.0 Tiny, because of unsupported architecture on +both oMLX and LMStudio. So much for modern models! ## New contenders @@ -80,7 +83,7 @@ than the [Gemma 4 E2B](https://artificialanalysis.ai/models/gemma-4-e2b/) (AAI 10) I tested last time. -With oMLX and a 8bits version, it runs using around 10GB +With oMLX and an 8-bit version, it runs using around 10GB unified memory. The first prompt is always a bit slow, but then the KV cache kicks in and it feels snappy enough. But after some time it ended up in a locked grep loop. I think @@ -90,7 +93,7 @@ I had an attempt on my desktop, which is better a prefill (input) than the Mac. But this time it didn't feel really smart. In the end, I'm a bit disappointed I couldn't run the newer models, -even though Ling 3.0 Tiny was supposed to fit. But maybe i can +even though Ling 3.0 Tiny was supposed to fit. But maybe I can run it later when the tooling has caught up. I think Qwen 3.5 4B may have a spot for offline use, as I can run it at 50 tok/s on Mac. I'd like to investigate using it |
