final draft

1 files changed, 143 insertions, 0 deletions

diff --git a/content/posts/machine-learning-formats/index.md b/content/posts/machine-learning-formats/index.md
new file mode 100644
index 0000000..902a325
--- /dev/null
+++ b/content/posts/machine-learning-formats/index.md
@@ -0,0 +1,143 @@
+---
+title:
+draft: true
+date: 2026-09-03
+lastmod:
+description:
+categories:
+  - Machine Learning
+---
+In 2019 I first got into tinkering with machine learning. I kind of dropped out of it at the end of the year, and now that it's all over the news, I feel I can't draw a direct connection to what I used to know. Today I'd like to try and bridge that gap.
+
+## The framework era
+
+The classic example work at the time was MNIST, hand written character recognition. This was cool because I think you could actually write the whole neural network in pure python, and still have it achieve reasonnable performance within tens of minutes of training.
+
+With today Torch framework, a simple solution looks like this simple module. It's a lot of jargon, but there is a single flow for data, and only two layers are stateful (`conv1` and `fc`) and actually learn. Those states are "the weights".
+
+```py
+class SimpleMNIST(nn.Module):
+    def __init__(self):
+        super()
+        # Input: single-channel image (28x28)
+        # 1. 2D convolution: 1 channel -> 8 channels, 3x3 filters
+        self.conv1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, padding=1)
+        # 2. Linear layer: 8 channels * 14x14 pixels -> 10 classes (digits 0 to 9)
+        self.fc = nn.Linear(8 * 14 * 14, 10)
+
+    def forward(self, x):
+        x = self.conv1(x)  # Shape: [batch, 8, 28, 28]
+        x = F.relu(x)  # ReLU activation
+        x = F.max_pool2d(x, 2)  # Shape: [batch, 8, 14, 14]
+        x = torch.flatten(x, 1)  # Flatten: [batch, 1568]
+        x = self.fc(x)  # Shape: [batch, 10]
+        return x
+```
+
+
+When I was doing research for my master's degree, each framework had a way to store and load weights for a specific module, or architecture. If you changed a step, you could not use the weights. This meant you basically had to share your script alongside your weights so someone could use it.
+
+## Runtimes and shared formats
+
+In 2019, the [ONNX](https://github.com/onnx/onnx) format became widely used. This format combines the architecture with the weights, so anyone can run a shared model. It comes with an official backend from Microsoft, [ORT - ONNX Runtime](https://github.com/microsoft/onnxruntime).
+
+To me, a simple definition of a runtime would be
+`Runtime: model, input → output`.
+If you remove the variable model, you end up with a more classic algorithm. In the same idea, Python is a runtime
+`Python: code, input → output`.
+The runtime itself can contain deterministic algorithms, and also provide resource management (memory allocation, thread scheduling, ...). Traditionnaly, in machine learning, pattern recognition benefits the most from using this trainable model slot.
+
+So you can basically use any training framework, export your trained model in ONNX, and a device with ORT can run the model in production. We can try this for ourselves.
+
+```py
+torch.onnx.export(
+    model,
+    torch.randn(1, 1, 28, 28),
+    "mnist_simple.onnx",
+    input_names=["input_image"],
+    output_names=["predictions"],
+    opset_version=18,
+)
+```
+
+Then, with the help of onnx lib, we can see the architecture is encoded.
+
+```py
+import onnx
+
+onnx_model = onnx.load("mnist_simple.onnx")
+onnx.checker.check_model(onnx_model)
+for i, node in enumerate(onnx_model.graph.node):
+    print(f"Node {i} : {node.op_type} (Input: {node.input} → Output: {node.output})")
+```
+
+```
+Node 0 : Conv (Input: ['input_image', 'conv1.weight', 'conv1.bias'] → Output: ['conv2d'])
+Node 1 : Relu (Input: ['conv2d'] → Output: ['relu'])
+Node 2 : MaxPool (Input: ['relu'] → Output: ['max_pool2d'])
+Node 3 : Reshape (Input: ['max_pool2d', 'val_4'] → Output: ['view'])
+Node 4 : Gemm (Input: ['view', 'fc.weight', 'fc.bias'] → Output: ['predictions'])
+```
+
+We can see the ORT defines kernels (Relu, Reshape, ...) that our model can use. If you compile a model with a recent ONNX but try to run it on an old kernel, the architecture would not be supported because not implemented. This allows programatic architectures while keeping high performant, hardware specific kernels.
+
+Any app can now use our model, but it needs to provide the pre and post processing - transforming an image in a vector, and out probabilities in characters.
+
+```py
+def load_image(path: str) -> np.ndarray:
+    # Preprocessing must mirror the training pipeline: grayscale, 28x28, [0, 1]
+    image = Image.open(path).convert("L").resize((28, 28))
+    pixels = np.asarray(image, dtype=np.float32) / 255.0
+    if pixels.mean() > 0.5:
+        # MNIST digits are white on black; invert dark-background images
+        pixels = 1.0 - pixels
+    return pixels.reshape(1, 1, 28, 28)
+
+
+def detect_char(session: ort.InferenceSession, image_path: str) -> int:
+    input_name = session.get_inputs()[0].name
+    logits = np.asarray(session.run(None, {input_name: load_image(image_path)})[0])
+    return int(logits[0].argmax())
+
+
+session = ort.InferenceSession(
+    "mnist_simple.onnx", providers=["CPUExecutionProvider"]
+)
+```
+
+## Specialized runtimes
+
+As you can see, pre and post processing can be a certain toil. ORT is a very general runtime, but for specific use cases, runtimes can provide a more complete harness.
+
+In Computer Vision - what we are doing here, we need image encoding, maybe resizing and normalisation. Actually the few lines we wrote could become a Computer Vision runtime. Our CLI could take any image in, do the transformation, and use the model of our choice as the core intelligence.
+
+In Speech Recognition, voice activity detection, token decoding,  FFT are steps that benefit little from learning, and instead benefit on highly optimized code - FFT is done best without matrix multiplication. This is why sherpa-onnx is, as its name suggest, a speech specific, hardcoded harness around a ORT pass.
+
+With LLMs, you hear way more often about GGUF than ONNX, and llama.cpp than ORT. LLMs needs tokenisation - the pre processing step, caching, quantization, parallelization, and more that I don't know.
+
+## Example: SqueezeNet, 1.3 MB for Computer vision
+
+I download the ONNX file, then I can ask a LLM to produce the fow chart using the onnx lib, or I can use [Netron](https://netron.app/) to display it. This vision model is simple enough to read: 65 nodes, only 7 distinct op types, a chain of convolutions. with some quantization.
+
+{{< details summary="Full SqueezeNet ONNX graph" >}}
+![SqueezeNet ONNX graph](squeezenet1.0-12-int8.onnx.svg)
+{{< /details >}}
+
+So I hope you get the picture now. Each use case has specific pre and post processing, and also processing steps that are hardcoded - and often not adapted to GPUs. In the middle, it loads matrix weights for large mutiplication - that's the part that learned. sherpa-onnx for speech, fastdeploy for Vision all use ORT for this middle ONNX compute. Other have recoded a new runtime to get even better performance - llama.cpp and the GGUF format supports very aggresive compression on weights for LLMs.
+
+Here is a small compilation of formats & runtimes couples, in case you hear about them:
+- ONNX + ORT - General purpose
+- ONNX + sherpa-onnx - Real time audio
+- GGUF + llama.cpp / ggml - LLM, memory optimized, compression
+- Safetensors + vLLM - FP16, mmap, datacenter LLM inference
+- TensorRT Engine (.engine / .plan) + NVIDIA Runtime - CUDA kernels, audio / vision
+- OpenVINO IR (.xml + .bin) + Runtime - Intel optimized
+- Core ML (.mlpackage) + Framework - Apple Silicon
+- RKNN + rknn-toolkit-lite - Rockchip / ARM Edge AI
+
+{{< alert info "LLM usage in this article" >}}
+- Research
+- Writing pytorch code samples
+- Exploring ONNX files structure
+- Acting as Publisher
+{{< /alert >}}