1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
|
---
title: ""
draft: true
date: 2026-08-22
lastmod:
description:
categories:
- LLM
- Coding Agents
---
## Qwen 3.8 27B
This week was interesting for local models, because we had the
open weights release of Qwen 3.8 27B. It was much expected, because
Qwen 3.6 held the crown for a long time in open models of a size
that is runnable on almost consumer hardware.
The [Artificial Analysis](https://artificialanalysis.ai/models/qwen3-8-27b)
score is pretty damn good! 52 with xhigh is in the ballpark of
GPT-5.6 Luna (max) and DeepSeek V4 Flash 0731, which were my go
to models for august, given their relatively low price.
To think that I could run as a powerful model locally feels
insane.
For reference, it sits between a Claude Opus 4.6 and a
Claude Opus 4.7, and that was the best what we had in February
in Claude Code.
## Artificial Analysis pareto frontier
https://artificialanalysis.ai is my reference for model comparison.
It actually compiles scores from many benchmarks. I know there are
many pitfalls to these benchmarks, but hey, we have to reduce
the world's complexity, right?
I found my prefered charts, and that's when you remove all models.
Then AAI highlights the pareto frontier, with the best option
at each tradeoff. Go see for yourself!
[Int vs Cost/Time per task](https://artificialanalysis.ai/models#intelligence-comparisons) - What frontier model to use in
what case. Here we can see Luna is pretty neat for cost, and
that's what I used in august.
[Int vs Active/Total params](https://artificialanalysis.ai/models#model-size) - This helps me understand what I could
run on my hardware. For this one, you need to filter on small
(and maybe tiny) models. This show Qwen 3.8 27B as much better
than the previous Gemma 4 26B A4B. And I can see a new contender
[Ling 3.0 Tiny](https://artificialanalysis.ai/models/ling-3-0-tiny)
that looks promising.
## Running modern models
I wonder if local inference got better since https://blog.kagamino.dev/posts/local-llm-inference-offline-scenario/.
Back then, I had decent inference performance with Gemma 4 E2B
(AAI 15), but it felt too dumb to do meaningful work.
I wanted to test Qwen 3.5 9B, but this time it was too slow,
on my Desktop as well as on my Mac.
As I ran out of my OpenCode Go subscription for the last ten days
of august, it's time to check if I can work locally!
Maybe there were new smaller and smarter models, like Qwen
or Ling ? Or better inference, like oMLX on Mac?
Last time was a one shot scenario, so this time I want to test
a coding agentic workflow with Pi - it has a smaller system prompt.
I'll go by feel, because I don't have clear metrics to target.
Sadly, Qwen 3.8 27B is too large, even with oMLX. That's also
because I need a larger context of 100k tokens. And I couln't
run Ling 3.0 Tiny, because of unsupported architecture. So much
for modern models!
## New contenders
But the AAI pareto frontier gives me new ideas of old models
to run.
[Qwen 3.5 4B](https://artificialanalysis.ai/models/qwen3-5-4b)
is still on this frontier with a AAI 20, and a smaller size
than the
[Gemma 4 E2B](https://artificialanalysis.ai/models/gemma-4-e2b/)
(AAI 10) I tested last time.
With oMLX and a 8bits version, it runs using around 10GB
unified memory. The first prompt is always a bit slow, but then
the KV cache kicks in and it feels snappy enough.
But after some time it ended up in a locked grep loop. I think
this is usable for targeted changes.
I had an attempt on my desktop, which is better a prefill (input)
than the Mac. But this time it didn't feel really smart.
In the end, I'm a bit disappointed I couldn't run the newer models,
even though Ling 3.0 Tiny was supposed to fit. But maybe i can
run it later when the tooling has caught up.
I think Qwen 3.5 4B may have a spot for offline use, as I can run
it at 50 tok/s on Mac. I'd like to investigate using it
as an autocomplete model for IDE.
|
