About +10% tokens/s generation speed for local AI
Two optimisations of the expert arena mode, shipped by the maintainer in release v0.1.44 with my authorship kept.
Show technical detailsHide details
Open-source inference engine that runs large MoE models (Qwen) on consumer hardware, with a local OpenAI- and Anthropic-compatible API.
- Async adaptive tier (#1904):
--adapt-async 1now works in arena mode on a single GPU. Expert copies no longer block the next verify window (25.9 → 21.6 ms). Result: 97.2 → 107.7 tok/s on an RTX 3090, faster on 12 of 12 prompts. The maintainer measured +4.2 to +5.2% on an RTX 5070 and +7 to +10% on code and document prompts. - Pin by need (#1905): on Windows, PCIe DMA registration only covers part of the arena.
STRATA_PIN_BY_NEED=1pins the experts the GPU does not hold first. Prompt read +5 to +7% at 56K–117K tokens, up to +8.5% measured by the maintainer. - Method: reproducible cold benchmarks, branch versus baseline, documented limits (gain is neutral after prompts longer than 56K tokens).











