Machine Learning
Strict now has working NeuralNetworks and MachineLearning packages. They exercise the same language, inline tests and native compiler as other programs. The neural-network layers, training algorithms, transformer blocks and tokenizers are written in Strict; small native helpers read tensor data.
Start small
From the Strict repository root:
strict build NeuralNetworks/Xor.strict
strict build NeuralNetworks/Iris.strict
strict build MachineLearning/IrisClassifiers.strict
Run the resulting executable next to each source file to try it. Small programs can also run on the VM with strict NeuralNetworks/Xor.strict.
| Example | What it demonstrates |
|---|---|
| LogicGates and Xor | Perceptrons and a hidden layer |
| Iris | Classification, softmax and cross-entropy training |
| ActivationComparison and OptimizerComparison | Activation functions, momentum, RMSprop and Adam |
| BinaryAddition and ReberPrediction | Recurrent networks and GRUs learning sequences |
| Neuroevolution and ArmEvolution | Evolving network weights and robot-arm joint angles |
| IrisClusters and IrisClassifiers | K-means, nearest neighbors, naive Bayes, decision trees and random forests |
| MarkovText | Character generation from a downloaded text corpus |
Larger datasets belong outside the compiler repository. The Mnist project contains the handwritten-digit example.
GPT-2 and Qwen3 inference
strict build NeuralNetworks/Gpt2Text.strict
strict build NeuralNetworks/Qwen3Text.strict
These examples download model files on first use: GPT-2 124M uses a roughly 548 MB safetensors file in gpt2/, and Qwen3 0.6B uses a roughly 805 MB Q8_0 GGUF file in qwen3/. Use native builds for the real models; the VM is not suitable for them. Inline tests use tiny model fixtures instead.
- GPT-2: safetensors loading, a byte-pair tokenizer, transformer blocks with cached keys and values, and greedy generation. Weights can use 32-bit floats, 16-bit halves or Q8_0 quantization through
Gpt2.Loaded(path, 32),16or8. - Qwen3: GGUF loading, RMS normalization, grouped-query attention, rotary positions and SwiGLU. The reader supports F32, F16, BF16, Q8_0, Q4_K and Q6_K tensors. Packed quantized blocks stay compact during execution. Mixture-of-experts routing supports Qwen3-Coder-30B-A3B as well as dense Qwen3 models.
- Shared generation:
Generationcontinues prompts with either model. Tests compare the tiny fixtures with reference logits and tokens from the transformers library.
Qwen3-8B: faster than llama.cpp
Qwen3-8B Q4_K_M reaches 150–151 generated tokens/second (6.6–6.7 ms per token) on an RTX 4090 in the development results reviewed October 19. The recorded llama.cpp b11658 CUDA comparison takes 7.25 ms (138 tokens/second). The transformer logic is written in Strict. The measured baseline is llama.cpp, not a separate Ollama run.
| Model and weights | Strict GPU decode | llama.cpp CUDA comparison |
|---|---|---|
| Qwen3-8B Q4_K_M | 6.6–6.7 ms/token (150–151 tokens/s) | 7.25 ms/token (138 tokens/s) |
| Qwen3 0.6B Q4_K_M | 2.2–3.1 ms/token | 2.2 ms/token |
| Qwen3-Coder-30B-A3B Q4_K_M (mixture of experts) | 4.76 ms/token (210 tokens/s) | 5.26 ms/token (190 tokens/s) |
| Qwen3.8 27B Q4_K_M (Gated DeltaNet hybrid) | 21.5 ms/token (46.6 tokens/s) | 23.2 ms/token (43.0 tokens/s) |
| Qwen3.8 Brilliance 50X Q4_K_M (reasoning) | 22.1 ms/token (45.3 tokens/s) | 23.0 ms/token (43.5 tokens/s) |
The recorded setup uses an RTX 4090 with 24 GB VRAM, a Threadripper 3990X and 128 GB RAM. The comparison uses the same GGUF weights and greedy sampling. Qwen3.8 27B with MTP drafting decodes at 13.7 ms/token (73 tokens/s, Ollama 13.3 ms). MossImage diffusion at 720x1280 takes 453-463 ms per step on an RTX 3080. These are steady-state decode measurements; prompt processing, context length, batching and startup are separate costs. In particular, the 30B example's initial upload of 18.6 GB of experts makes a short end-to-end run much slower than its 210 tokens/s decode rate. The NeuralNetworks README records the measurements; the benchmark reports warmup, prefill, median, mean and p95 decode latency separately.
Prompt processing, batches and longer context
Qwen3-8B now processes prompts in chunks of up to 2,048 tokens. Tiled quantized matrix products and FlashAttention-style attention bring prefill to 0.15 ms per token at both 460 and 1,820 tokens, and 0.17 ms at 3,720 tokens. These are the fastest of three prefills; full collections can interrupt individual runs. A 10-token prompt takes 1.0–1.2 ms/token and a 100-token prompt 0.21 ms/token. The recorded llama.cpp figures for 10, 100 and 460 tokens are 1.18, 0.175 and 0.10 ms/token.
| Qwen3-8B simultaneous sequences | Aggregate decode throughput | Recorded llama.cpp comparison |
|---|---|---|
| 2 | 285 tokens/s | 220 tokens/s |
| 4 | 530 tokens/s | 439 tokens/s |
| 8 | 953 tokens/s | 727 tokens/s |
| 16 | 1,655 tokens/s | — |
| 32 | 2,357 tokens/s | — |
These batch results combine tokens from all sequences at the same position, separate from single-sequence latency.
An optional 8-bit key/value cache uses 74 KB instead of 295 KB per token for the 8B model, fitting about four times the context in the same cache memory. Quantization can change generated text. With that cache, decode takes 8.6 ms/token after 7,450 prompt tokens, 9.7 ms after 14,900 and 12.8 ms after 29,700.
Build the benchmark from the repository root, then pass a local GGUF file:
strict build NeuralNetworks/Qwen3Benchmark.strict
NeuralNetworks/Qwen3Benchmark.exe path/to/Qwen3-8B-Q4_K_M.gguf 4 128
On Linux run NeuralNetworks/Qwen3Benchmark without the .exe suffix. The second argument keeps the weights in their quantized form; the final argument selects the number of generated tokens.
What makes it fast
Quantized matrices remain packed, numeric loops become GPU kernels, intermediate calculations stay on the device and graph replay reduces host launch overhead. CPU workers and SIMD remain useful for other workloads. Set STRICT_GPU=0 for a CPU comparison or STRICT_GRAPHS=0 to measure without graph replay. See Concurrency.
Graph replay, tiled tensor-core matrix products, fused bias operations, flash attention, batched decoding and the 8-bit cache are implemented and measured. Performance results are recorded per model and workload as the implementation evolves. The earlier GPT-2 CPU benchmark reached 8.6 ms/token with 8-bit weights on an i9-13900KF; it is a different model and setup from the Qwen3 GPU results above.
More than inference
The package also trains networks and implements recurrent models, GRUs, optimizers and evolutionary algorithms. Qwen3CoderText runs Qwen3-Coder-30B-A3B with expert routing. Tiny model fixtures compare logits and generated tokens with reference implementations, keeping the tests small while real models remain external downloads.
Open a block or layer in VS Code, press F12 into the operations it uses and read the tests beside the implementation. Line values and coverage make these algorithms easier to follow, using the same workflow as your own application code.
