StartLux Open-Sources Prism, a Decision-Model Inference Engine

Decision models and chat models do different jobs. A chat model writes tokens one by one. A decision model reads a state (text, JSON, or images) plus typed questions, then returns a calibrated probability for every listed option — choice, yes/no (noul), or a score on a scale — so software can threshold, log, and act without parsing free text. Instant variants answer in one forward pass; AutoThink variants do the same when confident, and write a short thought only when the top probability falls below a gate.

That difference matters for serving. Engines such as vLLM and SGLang are built around token generation: each question tends to become its own completion, the full vocabulary LM head is scored, and shared evidence is re-read. Prism’s README and design notes rebuild the path around “one distribution over options,” not “keep generating until EOS.”

What Prism changes in the forward path

Four ideas show up consistently across the Prism README, LeiPhone, and Pandaily:

• Score the options, not the dictionary. For the 27B model the full LM head is on the order of ~248k tokens. A decision only needs the lettered options (A–Z). Prism scores those letters at the final step; softmax over the listed options is mathematically the same as picking letters out of a full-vocab distribution.
• Forest / tree batching. Shared instructions are the root, shared material the trunk, each question a branch. One document with three questions shares the evidence pass instead of three independent HTTP/tokenize/prefill cycles.
• Deterministic int8. Projections run on int8 tensor cores with int32 accumulation, so the same request yields bit-identical probabilities across concurrency, GPU count, and prefix-cache hits — useful when a product action hinges on the last probability digits.
• Shared AutoThink scheduler. Direct answers and longer thoughts share one GPU scheduler (CUDA graphs; up to 64 think sequences on an H200 for 27B). Short new requests wait one or two decode steps rather than queue behind someone else’s long thought. Prefix cache: a follow-up on the same evidence is reported at 5.4 ms vs 54 ms without it (README figure).

Illustration: chat-style separate completions vs Prism’s shared-evidence forest. Speedups below are vendor-reported on one H200.

Vendor numbers on one H200 — keep the quote marks

StartLux publishes single-GPU comparisons of Prism int8 against the official StartLux-Decision engine on vLLM 0.31 FP8, using StartLux-Decision v1.1 27B AutoThink. Treat every millisecond as a company measurement on their harness:

• Short question (~256 tokens): 19.0 ms vs 53.1 ms (~2.8×)
• Same material, three questions: 17.4 ms vs 92.9 ms (~5.3×) — the gap that forest batching is meant to explain
• Accuracy: int8 92.94% vs BF16 92.97% on seven suites / 12,351 decisions (v1 evaluation referenced in README alt text / benches)
• JevBench (press / English summaries): Prism ~41 ms average vs SGLang 75.6 / vLLM 112.7 / llama.cpp 589 on 231 questions — again StartLux’s comparison set
• Scale: TP cuts single-question latency; DP raises throughput (README: four H200s ~121 read-outs/s; eight ~238). Under heavy concurrency the lead vs FP8 narrows because both hit tensor-core limits.

Press also cites a compressed ~16.55 GB build on a single 32 GB RTX 5090; the Prism README states that in 4-bit (W4A8 from Q4_K_M) the 27B runs on one 32 GB RTX 5090. Confirm the exact artifact size from the GGUF you download.

What it serves — and what it does not

Prism serves v1 and v1.1 AutoThink at 4B / 9B / 27B straight from released GGUF (Q8_0 as int8 / Q4_K_M as W4A8) behind the same /v1/systemone (also /v1/decisions) API used by the Decision package. Images need the matching mmproj file (up to eight per request on v1.1).

v1.1 Instant is not supported yet — Instant uses bidirectional attention over the prompt, and Prism’s table marks that row empty. Teams on Instant stay on the Decision package’s default vLLM path (or llama.cpp / MLX) until Prism adds it.

Honest caveats

• Vendor benches only — H200 latency/throughput and JevBench cross-engine numbers are StartLux’s; re-run with the README’s bench / eval commands on your hardware.
• Instant unsupported on Prism; AutoThink GGUF path only for now.
• CC BY-NC 4.0 weights — not a free-for-commercial model drop.
• Engine license in flux — prefer the live Prism README over press that called the engine Apache-2.0 wholesale.
• No TLS / API keys on the Prism server — treat it as an internal process.
• CUDA 13 (or documented 12) and a first-start compile — not a one-click cloud widget.
• This is the decision-stack cousin of the broader “model maker ships its own engine” trend (e.g. DeepSeek’s DeepInfer for its own models); Prism’s angle is the decision-model runtime, not a general chat accelerator.

Bottom line for overseas engineers already evaluating StartLux-Decision, or anyone building routers / policy checkers / multi-question ticket triage: if your workload is “one state, several typed questions, act on probabilities,” Prism is the open CUDA path StartLux wrote for that shape — and the 5.3× multi-question claim is exactly where forest batching should help most. Measure it yourself, keep Instant on vLLM for now, and read the NC weight license before you ship a product.