polish

1 files changed, 26 insertions, 21 deletions

diff --git a/content/posts/machine-learning-formats/index.md b/content/posts/machine-learning-formats/index.md
index 902a325..de7338e 100644
--- a/content/posts/machine-learning-formats/index.md
+++ b/content/posts/machine-learning-formats/index.md
@@ -1,9 +1,9 @@
 ---
-title:
+title: Machine Learning model formats and runtimes
 draft: true
-date: 2026-09-03
+date: 2026-09-04
 lastmod:
-description:
+description: The Machine Learning ecosystem devised general purpose file formats and runtimes to share compute graphs and weights, and solve deployment issues since the 2010s. Now hardware or domain specific runtimes are being added to provide improved capabilities.
 categories:
   - Machine Learning
 ---
@@ -11,9 +11,9 @@ In 2019 I first got into tinkering with machine learning. I kind of dropped out
 
 ## The framework era
 
-The classic example work at the time was MNIST, hand written character recognition. This was cool because I think you could actually write the whole neural network in pure python, and still have it achieve reasonnable performance within tens of minutes of training.
+The classic work at the time was MNIST, hand written character recognition. This was cool because I think you could actually write the whole neural network in pure python, and still have it achieve reasonable performance within tens of minutes of training.
 
-With today Torch framework, a simple solution looks like this simple module. It's a lot of jargon, but there is a single flow for data, and only two layers are stateful (`conv1` and `fc`) and actually learn. Those states are "the weights".
+With today's Torch framework, a simple solution looks like this simple module. It's a lot of jargon, but there is a single flow for data, and only two layers are stateful (`conv1` and `fc`) and actually learn. Those states are "the weights".
 
 ```py
 class SimpleMNIST(nn.Module):
@@ -34,7 +34,6 @@ class SimpleMNIST(nn.Module):
         return x
 ```
 
-
 When I was doing research for my master's degree, each framework had a way to store and load weights for a specific module, or architecture. If you changed a step, you could not use the weights. This meant you basically had to share your script alongside your weights so someone could use it.
 
 ## Runtimes and shared formats
@@ -45,9 +44,9 @@ To me, a simple definition of a runtime would be
 `Runtime: model, input → output`.
 If you remove the variable model, you end up with a more classic algorithm. In the same idea, Python is a runtime
 `Python: code, input → output`.
-The runtime itself can contain deterministic algorithms, and also provide resource management (memory allocation, thread scheduling, ...). Traditionnaly, in machine learning, pattern recognition benefits the most from using this trainable model slot.
+The runtime itself can contain deterministic algorithms, and also provide resource management (memory allocation, thread scheduling, ...). Traditionally, in machine learning, pattern recognition benefits the most from using this trainable model slot.
 
-So you can basically use any training framework, export your trained model in ONNX, and a device with ORT can run the model in production. We can try this for ourselves.
+So you can basically use any training framework, export your trained model in ONNX, and a device with ORT can run the model in production. I can try this for myself.
 
 ```py
 torch.onnx.export(
@@ -60,7 +59,7 @@ torch.onnx.export(
 )
 ```
 
-Then, with the help of onnx lib, we can see the architecture is encoded.
+Then, with the help of onnx lib, I can see the architecture is encoded.
 
 ```py
 import onnx
@@ -79,9 +78,11 @@ Node 3 : Reshape (Input: ['max_pool2d', 'val_4'] → Output: ['view'])
 Node 4 : Gemm (Input: ['view', 'fc.weight', 'fc.bias'] → Output: ['predictions'])
 ```
 
-We can see the ORT defines kernels (Relu, Reshape, ...) that our model can use. If you compile a model with a recent ONNX but try to run it on an old kernel, the architecture would not be supported because not implemented. This allows programatic architectures while keeping high performant, hardware specific kernels.
+I can see the ONNX format defines ops (Relu, Reshape, ...), and ORT implements kernels that the model can use. If you export a model with a recent ONNX but run it on an old runtime, the architecture would not be supported because not implemented. This allows programmatic architectures alongside highly performant, hardware specific kernels.
+
+This is not hypothetical: in [my recent tests around local LLMs](/posts/new-tests-around-local-llms/), llama.cpp refused to run Ling 3.0 Tiny, because of an unsupported architecture. The GGUF file declares `bailingmoe3`, a hybrid design that was brand new. The individual bricks already existed in llama.cpp, but each architecture needs its own explicit graph implementation - this one landed on August 17, 2026 ([PR #26608](https://github.com/ggml-org/llama.cpp/pull/26608)), right after my tests. Same story as ONNX: the format carries the declaration, the runtime implements it.
 
-Any app can now use our model, but it needs to provide the pre and post processing - transforming an image in a vector, and out probabilities in characters.
+Any app can now use the model, but it needs to provide the pre and post processing - transforming an image into a vector, and output probabilities into characters.
 
 ```py
 def load_image(path: str) -> np.ndarray:
@@ -107,24 +108,22 @@ session = ort.InferenceSession(
 
 ## Specialized runtimes
 
-As you can see, pre and post processing can be a certain toil. ORT is a very general runtime, but for specific use cases, runtimes can provide a more complete harness.
+As you can see, pre and post processing can be tedious. ORT is a very general runtime, but for specific use cases, runtimes can provide a more complete harness.
 
-In Computer Vision - what we are doing here, we need image encoding, maybe resizing and normalisation. Actually the few lines we wrote could become a Computer Vision runtime. Our CLI could take any image in, do the transformation, and use the model of our choice as the core intelligence.
+In Computer Vision - what I'm doing here, I need image encoding, maybe resizing and normalization. Actually the few lines I wrote could become a Computer Vision runtime. The CLI could take any image in, do the transformation, and use the model of my choice as the core intelligence.
 
-In Speech Recognition, voice activity detection, token decoding,  FFT are steps that benefit little from learning, and instead benefit on highly optimized code - FFT is done best without matrix multiplication. This is why sherpa-onnx is, as its name suggest, a speech specific, hardcoded harness around a ORT pass.
+In Speech Recognition, voice activity detection, token decoding, FFT are steps that benefit little from learning, and instead benefit from highly optimized code - FFT is done best without matrix multiplication. This is why [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) is, as its name suggests, a speech specific, hardcoded harness around an ORT pass.
 
-With LLMs, you hear way more often about GGUF than ONNX, and llama.cpp than ORT. LLMs needs tokenisation - the pre processing step, caching, quantization, parallelization, and more that I don't know.
+With LLMs, you hear way more often about GGUF than ONNX, and llama.cpp than ORT. LLMs need tokenization - the pre processing step, caching, quantization, parallelization, and more that I don't know.
 
-## Example: SqueezeNet, 1.3 MB for Computer vision
+## Example: SqueezeNet, 1.3 MB for Computer Vision
 
-I download the ONNX file, then I can ask a LLM to produce the fow chart using the onnx lib, or I can use [Netron](https://netron.app/) to display it. This vision model is simple enough to read: 65 nodes, only 7 distinct op types, a chain of convolutions. with some quantization.
+Let's take a small vision model and use [Netron](https://netron.app/) to display its architecture. [SqueezeNet](https://github.com/forresti/SqueezeNet) was released in 2016.
 
 {{< details summary="Full SqueezeNet ONNX graph" >}}
 ![SqueezeNet ONNX graph](squeezenet1.0-12-int8.onnx.svg)
 {{< /details >}}
 
-So I hope you get the picture now. Each use case has specific pre and post processing, and also processing steps that are hardcoded - and often not adapted to GPUs. In the middle, it loads matrix weights for large mutiplication - that's the part that learned. sherpa-onnx for speech, fastdeploy for Vision all use ORT for this middle ONNX compute. Other have recoded a new runtime to get even better performance - llama.cpp and the GGUF format supports very aggresive compression on weights for LLMs.
-
 Here is a small compilation of formats & runtimes couples, in case you hear about them:
 - ONNX + ORT - General purpose
 - ONNX + sherpa-onnx - Real time audio
@@ -135,9 +134,15 @@ Here is a small compilation of formats & runtimes couples, in case you hear abou
 - Core ML (.mlpackage) + Framework - Apple Silicon
 - RKNN + rknn-toolkit-lite - Rockchip / ARM Edge AI
 
-{{< alert info "LLM usage in this article" >}}
+I hope you get the picture. Each use case has specific pre and post processing, and also processing steps that are hardcoded - and often not adapted to GPUs. In the middle, it loads matrix weights for large multiplication - that's the part that learns. sherpa-onnx for speech, [fastdeploy](https://github.com/PaddlePaddle/FastDeploy) for Vision all use ORT for this middle ONNX compute. Others have recoded a new runtime to get even better performance - llama.cpp and the GGUF format support very aggressive compression on weights for LLMs.
+
+{{<alert info "LLM usage in this article">}}
 - Research
 - Writing pytorch code samples
 - Exploring ONNX files structure
 - Acting as Publisher
-{{< /alert >}}
+{{</alert>}}
+
+{{<alert info "Git tips">}}
+You can see typo changes more easily with `git diff --word-diff`. This changes my life!
+{{</alert>}}