About the numbers in the title: 37 tok/s is with drafts from the model’s own MTP block, which the model checks itself, so the text is exactly the one it would have written without them. Without drafts, the like-for-like comparison with llama.cpp is 24 against 2.8 tok/s.
On a laptop with 32 GB of RAM (Intel Core Ultra 9 185H, integrated Arc GPU, NVMe SSD, Debian 13), Janas-LLM runs Qwen3-Next-80B-A3B Q4_K_M, a 48.4 GB file, larger than the memory:
- writing: 37 tok/s with drafts, 24 without; llama.cpp 2.8;
- reading a 1,024-token prompt: 62 tok/s against 16.
Try it now: https://github.com/prabanta-dev/janas/releases/latest
On the other 12 models compared, from Qwen3-0.6B to Qwen3.6-35B-A3B and Gemma 4, Janas-LLM was faster than llama.cpp at writing on every one, and at reading on all but one. The exception is unsloth’s 3-bit Qwen3-30B-A3B, which reads a prompt at 100 tok/s against 112. The full tables and how they were measured are in the README.
Janas-LLM is an inference engine written in C with no dependencies beyond the C library, built for mixture-of-experts models larger than the memory. Here is how it works.
The JNS format. A GGUF file is rewritten, not re-quantized. Every expert of every layer gets a slot of its own, with gate, up and down back to back, so the engine reads an expert with a single aligned O_DIRECT request. Everything else goes into a resident region. A dense model is a mixture with one expert and no router, so the same path serves both. The conversion is deterministic: the same GGUF gives the same bytes on any machine, and MODELS.md publishes the SHA-256 chain from the GGUF to the JNS.
The expert cache. The experts a token needs are read from the NVMe withO_DIRECTandio_uring, bypassing the page cache; the ones that keep coming back stay in memory. On the 80B the cache grows to about 19 GB, and the rest is read from disk. On the same model, llama.cpp’s usual build ran out of memory as soon as the GPU took part. It ran only with a build without repacking (GGML_CPU_REPACK=OFF) and with the experts memory-mapped (--no-host 1 -lm mmap), that is, read through the page cache.
Experts read ahead. When the cache is too small, the next layer’s router is applied early, and the experts it picks that are not in memory are read while the current layer computes. A long prompt is read the same way, block by block. With a 4 GiB cache, the 80B reads 4,096 tokens at 42.0 tok/s instead of 36.8. No result changes.
Light experts not in memory are skipped. This is the cache-conditional routing of Skliar et al. (TMLR 2025), applied only when the cache holds less than a fifth of the experts. An expert the router weighs lightly, and that would have to come from disk, is neither read nor computed, and its share goes to the token’s other experts. On the 80B with a 4 GiB cache, 20.7 tok/s become 27.7. Agreement with llama.cpp’s most likely token, over 400 tokens, goes from 393 positions to 384.
The kernels. Q4_K, Q6_K, Q8_0, Q3_K, IQ4_XS, IQ3_S and the other types share a single kernel body for 1-8 vectors, compiled for AVX2 and for AVX-VNNI. For a reply’s tokens every path gives bit-identical results, the GPU included. On 12 threads Q4_K runs at 69 GB/s and Q8_0 at 84, against the 81 GB/s the memory actually delivers.
The GPU is optional. The Vulkan 1.3 backend is used only where the engine measures that it helps. A prompt goes to the GPU’s tiled products. On the dense Qwen3 and Qwen3.5 models the whole block, attention and recurrent layers included, goes to the GPU in one submission while the CPU rests: on a laptop, a CPU working beside the GPU takes power the GPU would use. Qwen3-4B reads a prompt at 220 tok/s, against 85 without the GPU. These sums are not bit-identical to the CPU’s, but against the references the most likely token agrees just as often: within 2-3 positions out of 256.
Verified drafts. The text is exactly the one the model would have written on its own. The drafts come from the model’s MTP block where it has one (for the dense Qwen3.5 models, hf2jns_mtp converts it from the original checkpoint), from Gemma 4’s assistant, or from the conversation itself. In a chat reply, Qwen3.6-35B-A3B goes from 25 to 34 tok/s.
Small Gemma 4. E2B’s per-layer embeddings, a table as large as half the file, are read from the file one row at a time and never loaded. E2B ran in 1.5 GB of memory from a 3.1 GB file.
On Hugging Face there are 14 repositories under prabanta-dev, grouped in the collections “Qwen for Janas-LLM” and “Gemma 4 for Janas-LLM”. They hold the models already converted to JNS, with the original licence, the attributions, the source quantizer and the SHA-256 of every file. The JNS v2 format will be readable by every future version. janas-get downloads and verifies them.
How to try it, on Ubuntu 22.04+ or Debian 12+. Download the package from https://github.com/prabanta-dev/janas/releases/latest , or:
curl -LO https://github.com/prabanta-dev/janas/releases/latest/download/janas-linux-x86_64.tar.gz
tar xf janas-linux-x86_64.tar.gz && cd janas
./janas-chat
janas-chat lists the models it knows: pick one, and it is downloaded and checked. Besides the chat there are a C library with FreeBASIC bindings, an HTTP server whose API follows OpenAI’s, and an MCP server.
A Windows version is in the works.
The limits, plainly. It is an experiment, developed and measured on a single machine. For now it runs on Linux x86-64 only. The API changes without notice; the JNS files do not.
What would help most: janas-bench reports from other hardware, above all CPUs without AVX-VNNI, AMD, discrete GPUs, machines with 16 GB or less, and slower disks. The engine picks threads, cores and GPU use from what it measures, but so far it has measured only one machine.
- Release: https://github.com/prabanta-dev/janas/releases/latest
- Code (GPL-3.0-or-later): https://github.com/prabanta-dev/janas
- Documentation: https://github.com/prabanta-dev/janas/wiki
Maurizio “camauri” Cammalleri