Turning the Community-Built Inference Engine “Strata” into a Single-GPU Inference Appliance

Introduction

Qwen3.8-Flash-Next is a model that delivers response quality good enough that I’d almost call it a next-generation model that has already reached a certain level of completion. Because of that, quite a few people want to get their hands on it — popular enough that some people have gone out of their way to buy up multiple used NVIDIA Tesla V100 32GB cards just to run it.^[1]

Figure 1: Architecture diagram of Qwen3.8-Flash-Next

This model’s parameter count comes out to 125B + 51B (n-gram) + 4B (MTP) – A6B — a fairly large model. (Of that, the 51B n-gram portion, roughly 27GB, runs out of main memory.) When I evaluated it myself, my own environment couldn’t handle it at all, so I borrowed a machine with an NVIDIA A100 from Highreso to run the evaluation. Then, right around early October, I heard that this model could be run with far fewer resources, and still run fast.

My first reaction was, “No way that’s real — it’s probably just some trick that fakes a speed boost,” but…

What I’m introducing this time is Strata, an inference engine that actually pulls this off.

How to read this article This article walks through how Strata works and shares measured results covering KV/Expert cache behavior. If you’d like to read it in order, just keep going. If you just want the numbers, feel free to skip ahead to “What’s It Actually Like to Use?”

About Strata

Strata is an inference engine built on top of llama.cpp, packed with various tricks that let it run quantized versions of Qwen3.8-Flash-Next in low-resource environments. It can expose basically the same interface as llama.cpp, and it runs models from GGUF files. As a minimum setup, it needs:

• At least 48GB of free main memory

• At least 12GB of free VRAM

• At least 100GB of SSD space

— and with that, remarkably, it can run Qwen3.8-Flash-Next.

It’s created by Niko Veit (https://x.com/coldniko), built together with a group of volunteers. As of this writing, the latest version is v0.2.2.^[2]

Let me walk through each of the ideas built into it.

The 3-Tier Resource Structure

Rather than placing everything on the GPU, Strata lays resources out across a 3-tier structure, built to take good advantage of Qwen3.8-Flash-Next’s MoE architecture and its n-gram Embedding.

Figure 2: The 3-tier resource hierarchy

Only the parts that are absolutely required stay fixed on the GPU, while all non-shared Experts are placed in main memory. Frequently used Experts are then pulled into an “Expert Cache” as needed, so that only the highest-frequency Experts occupy the fast tier.

The KV Cache is likewise split across two tiers.

To squeeze out context length more efficiently, Strata adopted the Elastic KV approach (--kv-grow) starting in v0.1.40. Previously, the KV cache was a fixed size, overflowing into pinned host memory as a fallback when needed. With Elastic KV, the mechanism changed so that memory can be handed over from the cold Expert Cache to the KV cache first, when needed — allowing it to keep responding quickly even when longer context sizes are requested.

The n-gram Retriever, which Qwen3.8-Flash-Next originally keeps in main memory, gets shifted onto the SSD instead. In a sense, this is closer to the original intended use of PLE, and since that data is read directly from the GGUF file, it’s designed to avoid adding SSD overhead — a genuinely helpful touch in a resource-constrained environment.

Not Just a llama.cpp Wrapper

The engine that Strata runs on is built on top of ggml-org/llama.cpp, but it’s really a hybrid engine — a derivative engine with its own extensions — that adds substantial custom implementation on top for things like MoE offloading, speculative decoding, and custom kernels.

Automatic Tuning

At the end of the install process, Strata runs automatic tuning and sets reasonably optimal parameters for your environment. That means you can get solid performance without fine-tuning parameters by hand after installation, cutting down the time it takes to go from setup to production.

MTP Is On by Default

As a way to boost generation speed, the MTP (speculative token prediction) feature is on by default. This allows very fast token generation, especially in multi-turn scenarios, letting you fit more inference work into a limited amount of time.

The Load on the SSD Is Almost Entirely Read Load

Since Strata also uses the SSD, I was initially worried that something like an SSD cache might wear down the NAND through heavy rewriting. In practice, though, most of what occupies that space is n-gram Embedding. The data making up the n-gram Embedding is static knowledge data, so it never gets rewritten — it simply gets read as needed.

So the risk of the SSD degrading rapidly and becoming unusable because of Strata is expected to be extremely low.

A GPU Profile That Fits the Bill Perfectly

For this evaluation I used an easy-to-rent, NVIDIA RTX A4000-equipped GPU cloud instance.

It’s a container-based instance I’ve used often for evaluating things in an isolated environment, and for testing models an RTX 3060 can’t handle, before I eventually bought my own Intel ARC Pro B70.

Table 1: Specs of the NVIDIA RTX A4000 GPU instance used in this testing

Right as I was about to evaluate Strata, I looked at this machine’s specs and noticed they lined up almost perfectly with what Strata needs. So I decided to see whether I could turn this RTX A4000 instance into a proper appliance for running Qwen3.8-Flash-Next.

Running Strata

Starting Strata

To start Strata, run the launch command with the path you specified during installation. The first run went through the install process, but once a configuration already exists, startup proceeds according to that configuration.

$ /home/user/.local/bin/strata launch

The screen below appears; select the model name shown at the top and press Enter to start the engine. (Screen 1)

Note that the engine takes roughly 480 seconds to start up. During that time it’s loading and placing various things from disk into memory, so it’s worth being aware that this puts a fair amount of load on disk I/O.

Stopping Strata

To stop Strata, press Ctrl+C to terminate the process. Doing so brings up a screen like the one below.

(Screen 1)

Selecting Stop serving in the background stops it.

You can also just detach from the foreground and keep the engine running in the background — in that case, select Keep serving in the background.

Updating Strata

Strata has an update command that lets you update to the latest version. Below is an example run — you can see the engine and the control plane each getting updated.

$ strata update
Strata update (https://github.com/Niko1221/Strata/releases/latest/download)
Reading the signed update feed
downloading engine 0.1.48 linux-x86_64-cuda-cuda13: 100% (578 of 578 MiB)
Unpacking 0.1.48 linux-x86_64-cuda-cuda13
Checking the package signature and every file
Test-starting engine 0.1.48 linux-x86_64-cuda-cuda13 before switching to it
Switching to engine 0.1.48 linux-x86_64-cuda-cuda13
Engine 0.1.48 is installed and will be used on the next start
downloading Control 0.1.49.1: 100% (94 of 94 MiB)
Unpacking Control 0.1.49.1
Test-starting Control 0.1.49.1
Feed sequence 111, signed by 5d1bc23abbeffd0d
Engine 0.1.47 -> 0.1.48 updated (tested: started, answered one token, stopped)
Control 0.1.47 -> 0.1.49.1 updated. The next `strata` command already runs it; the app switches on its next start
Saved tuning presets: marked for the short re-check (nothing was deleted).
The previous engine version is kept; `strata components rollback` returns to it.

It looks like a functional test runs whenever the engine gets swapped out, so updates tend to take a while.

Configuring Strata

This is one of Strata’s weak points: there’s no proper reference documentation for its configuration. Also, the spec seems to differ slightly between the v0.1.x and v0.2.x series.

The configuration file lives at the following location.

/home/user/.config/strata/launch-presets/iq3_xxs-128k.json

In my case, the configuration file looked like this.

{
"access": "lan",
"api_key": "XXXXXXXXXXXXXXXXXXXXXXX",
"context": 131072,
"created": "2026-10-23 13:54:09.666900",
"data_dir": null,
"extra": {
"STRATA_PIN_BY_NEED": "on"
},
"family": "qwen",
"gpu_roles": null,
"gpus": null,
"kv": null,
"last_used": "2026-10-23 15:28:25.785562",
"model": "IQ3_XXS",
"model_path": null,
"mtp": null,
"name": "IQ3_XXS 128K",
"no_key": true,
"parallel": null,
"port": 8080,
"schema": "strata.launch/1",
"tune": "quick",
"tune_allow_small": false,
"tuned_at": "2026-10-23",
"tuned_preset": "4082656e867d420899b7348cd4987f74",
"vision": "cpu"
}

As for what each of these settings changes, here’s a table covering the fields that are actually present.

Table 3: Configuration fields and their details

Other launch-time arguments are covered in the appendix at the end of this article.

How Strata Works Internally

Running the strata command appears to kick off the surrounding processes in roughly the following flow.

Figure 4: The chain of components the strata command sets in motion

The strata command acts as a wrapper around ~/.local/share/strata/terminal/0.1.47/strata.py, which runs through a Python 3.12 build prepared specifically for Strata. strata.py then runs ~/.local/share/strata/terminal/0.1.47/setup.py, the launch command for the strata-terminal subcomponent.

As it happens, this same setup.py was originally the strata-engine launch command back in the v0.1.x series. The strata command introduced in v0.2.x is, in essence, a version with expanded management commands layered on top.

From there, setup.py launches a subcomponent called strata-engine, tailored to the environment, which runs as the actual engine itself. Tracing how the command changes at each step makes it easy to see exactly what configuration it ends up launching with.

Example: the input command

strata launch
--preset IQ3_XXS 128K
--no-tui

What gets launched first

/home/user/.local/share/strata/terminal/0.1.47/python/bin/python3.12 /home/user/.local/share/strata/terminal/0.1.47/strata.py launch
--preset IQ3_XXS 128K
--no-tui

What gets launched next

/home/user/.local/share/strata/terminal/0.1.47/python/bin/python3.12 /home/user/.local/share/strata/terminal/0.1.47/setup.py
--yes
--no-start
--family qwen
--model IQ3_XXS
--context 262144
--vision cpu
--experimental-speed-projection off
--data-dir /home/user/Strata-data

What gets launched last

/home/user/.local/share/strata/components/engine/0.1.47-linux-x86_64-cuda-cuda13/strata
--serve
--pack /home/user/Strata-data/packs/iq3_xxs
--native /home/user/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
--ple-gguf /home/user/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf
--expert-profile /home/user/.local/share/strata/terminal/0.1.47/data/expert-profile.bin
--expert-cache auto --prefill auto
--spec 4
--spec-min-p 0.5
--mtp /home/user/Strata-data/mtp/rt
--max-context 262144
--kv int8
--vision

The Structure of the Strata-data Directory

You probably noticed on screen that a Strata-data directory gets created during installation. What’s actually stored in there? Here’s what it looks like.

It holds everything related to the model — the model itself, the tokenizer, and so on. A profile gets created based on the model name chosen at install time, and everything is stored according to that profile.

Figure 5: Layout of the Strata-data directory

Strata doesn’t simply reconstitute the model straight from the GGUF file — the data actually goes through the process shown below before it’s stored.

Table 4: How data is read in, depending on category

Strata’s Web Frontend

llama.cpp comes with a web frontend, where hitting the URL directly shows a chat screen — and Strata has something similar built in too.

Figure 6: Strata’s web frontend

What really stands out is the monitoring screen, which you get to by clicking “Monitor” in the upper left of the screen.

Figure 7: The menu-switching area

This screen shows real-time status, and the “Model State” view in particular lets you see exactly what the model currently loaded on Strata is doing at any given moment.

You can even track whether the GPU or the CPU is doing the processing, down to hybrid states between the two.

Figure 8: The top half of the monitoring screen shows a range of real-time statuses

The bottom half of the monitor screen is laid out to show how much of the context is in use and how many Experts are resident in VRAM. On the right side, you’ll find a record of the inference work carried out so far.

Figure 9: The bottom half of the monitor screen

Note that if you have the API enabled, these features require you to properly set an API key.

(Screen 1)

Looking at the upper right of the screen, you’ll see it says “API key needed.”

In that case, scroll down a bit on the About screen, where you’ll find a field for entering the API key — set it there.

(Screen 2)

Once the display in the upper right changes to idle, authentication succeeded.

What’s It Actually Like to Use?

How Much Resource Does It Consume?

Comparing resource consumption across each tier gave the following results.

Table 5: Summary of resource consumption. Because a 3-bit quantization was used, the footprint stayed relatively small.

That left roughly 23,000 MiB of free memory and about 17 GiB of free storage, so running any additional tools on top of this might be a bit of a stretch. That said, if you’re running this as a dedicated appliance, I don’t think that constraint matters all that much.

For example, I think a setup like the one below would work well for using this inference environment across a workgroup or within an organization.

Figure 10: A configuration envisioned for enterprise use

How Good Is the Performance?

Here are the actual results from interacting with the model through a Dify Agent.

Table 6: Performance measurements from running Qwen3.8-Flash-Next on Strata

Honestly, the prefill speed isn’t all that fast. Prefill performance heavily reflects the raw speed of the GPU processor itself, and here the NVIDIA RTX A4000’s relative weakness shows through. The KV cache also isn’t held entirely in VRAM — not even at the 128k I configured here, let alone the model’s maximum of 256k. Only a fixed amount is kept in VRAM, with whatever doesn’t fit spilling into pinned main memory by design, which may have introduced some overhead there. That said, Strata’s prompt cache absorbs most of that, so the perceived speed doesn’t come across as especially slow.

What’s really surprising is just how high the hit rate is on the VRAM Expert cache. This ties into both the Expert and KV caches, and it’s precisely because that hit rate is so high that actual throughput comes out as high as it does, I’d guess.

As for the output speed itself — when I evaluated Qwen3.8-Flash-Next on an A100 instance a while back, llama.cpp’s output speed came out to roughly 50–60 tok/s, so seeing numbers beat that here was genuinely surprising.

Conclusion: Remember, This Is a Specialized Inference Engine

That said, because this setup — built around Qwen3.8-Flash-Next — is a specialized inference engine, trying to use it for inference with other models runs into some real strain.

• It only pays off when MoE and n-gram Embedding work together. This engine achieves its performance through a very finely structured, large-scale set of Experts, combined with shifting the n-gram Embedding — which would normally sit in main memory — onto the SSD via PLE. That means running something like a Dense model on it would be a real stretch. Move the Dense layer into main memory and output speed collapses into single-digit tok/s, dragged down by that throughput; move it into VRAM instead and you defeat the whole point.

  On top of that, Qwen3.8-Flash-Next is a model that uses N-gram Embedding to fill in the knowledge gap that would otherwise remain if it were built purely out of MoE. It’s not an exaggeration to say the whole scheme only works because that gets shifted onto the SSD via PLE. It might be implementable with a standard MoE architecture too, but MoE itself tends to lag behind Dense models on answer quality, so it hasn’t been widely adopted elsewhere — and applying Strata to other MoE models honestly feels like it would be more complexity than it’s worth.

• It still leans heavily toward individual use. The design assumes a prompt cache is in play, so disabling it is likely to hurt prefill performance. It’s also still early in development and, if anything, feels reminiscent of older versions of llama.cpp. My honest impression is that it’s still got a fair number of rough edges before it’s ready for concentrated enterprise use.

  That said, settings to turn Prompt Cache off and to configure Parallel have since been added, tying in with some of llama.cpp’s own features to make it hold up better for enterprise use — and wasted processing is steadily being trimmed away, making it run noticeably faster.

This isn’t limited to what I’ve covered here — it’ll all depend on the details of whatever models come next — but if Strata ever manages to absorb models other than Qwen3.8-Flash-Next into main memory this cleanly, it would overturn a lot of assumptions we currently take for granted. In that sense, I’m looking forward to seeing where this goes.

Personally, I get the sense this product was built with exactly that kind of foresight in mind, and I’m excited to see what comes next.

Notes

1. By the time this article is published, Qwen4.0 may already be out — but even accounting for that, Qwen3.8-Flash-Next remains a remarkably capable model. ↩︎
2. Development is moving very fast, so the actual version may well be further ahead by the time you read this. Thanks for your understanding. ↩︎
3. The advertised spec said 50GB, but the top command showed 64,044.5 MiB. My guess is the advertised figure was actually showing remaining memory. ↩︎

References

• Deepwiki, Niko1221/Strata https://deepwiki.com/Niko1221/Strata

• Niko1221. “Strata README.md (Version 0.2.2 / engine 0.1.47).” GitHub main branch. https://github.com/Niko1221/Strata/blob/main/README.md

• Niko1221/Strata. “strata_cli/launch/settings.py (LaunchSettings, SCHEMA, ACCESS_KINDS, split_setting, as_argv_changes, flags_text, slug).” GitHub main branch. https://github.com/Niko1221/Strata/blob/main/strata_cli/launch/settings.py

Appendix: strata launch Help (English reference)

Usage (as in the original)

usage: strata launch [-h] [--preset NAME] [--model NAME] [--family NAME]
[--model-path PATH] [--data-dir DIR] [--move-data]
[--context SIZE] [--kv FORMAT] [--vision off|on|gpu|cpu]
[--parallel N] [--mtp on|off] [--gpu LIST]
[--gpu-roles LIST] [--tune quick|thorough|skip] [--tune-allow-small]
[--set KEY=VALUE] [--pool-cores LIST]
[--access local|lan|cloudflare|tailscale] [--port N] [--api-key KEY]
[--api-key-env NAME] [--new-key] [--no-api-key]
[--save-preset NAME] [--retune] [--no-tui] [--open] [--yes] [--plan]
[--feed URL|DIR] [--output {human,json}][--list-presets]
[--delete-preset NAME] [--status] [--stop]
[--show-key]

Overview

Brings model setup, tuning, starting the serve, and monitoring together in one place.

Run it without any flags and a guided screen opens. You can choose to start from a saved preset or set up from scratch. Each screen asks you to decide just one thing.

Add flags and it does the same thing without asking questions, then opens the monitor. Add --no-tui and it prints the address and exits instead.

Presets are shared with the Strata app’s profiles and tuning presets.

Common

Table 7: Common options

What to Run

Table 8: Arguments needed for basic configuration

Who Can Use It

Table 9: Access-related settings

Running / Saving

Table 10: Options related to execution and saving settings

Manage

Table 11: Management options

Examples (commands kept as in the original)

Open the guided screen

(From the list of saved presets, or set up from scratch.)

strata launch

Start a saved preset and open the monitor.

strata launch --preset "IQ3_XXS 128K"

Specify everything on the command line, serve, and print the address.

strata launch --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \
--access lan --port 8080 --save-preset "Office" --no-tui

Use a model file you already have (a folder, or a single .gguf).

strata launch --model-path /data/models/IQ3_XXS --no-tui

Change just one advanced setting for this particular launch.

strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui

Change nothing — just show what would happen.

strata launch --plan --model q2_0

Manage presets and running state.

strata launch --list-presets | --delete-preset NAME | --status | --stop [--preset NAME]

List the apps Strata can connect to (the connect subcommand, not launch).

strata connect

Write OpenCode configuration for the running server (a backup is kept).

strata connect opencode --write

Appendix: List of Selectable Models

Table 12: Models selectable in the installer and the resources they require