native inference, retrieval, and agents for real-time engines
A C++17 inference runtime for applications with frame budgets. Run local or remote models, batch active conversations, retrieve native memory, constrain output, and stream results through one C ABI.
pre-1.0 · Apache-2.0 · read the architecture story
Release suites run on x86-64 Linux and Apple Silicon. Sanitizer suites cover concurrency, memory, and inference paths; the Unreal integration exercises real CPU generation, lifecycle, embeddings, containers, and the sample project.
Astral gives games and native applications one inference API. Local GGUF models, remote providers, continuous batching, embeddings, structured output, and cancellation share the same request model. Engine code polls or waits at its own boundary; token hot paths never call into application code.
Agents keep prompts, summaries, retrieved context, and chat history in native memory. A completed agent can release its decode slot without losing that state, then continue later. The embedded index supplies memory without adding a separate database service to the application.
Astral ships no model weights. It is the native layer between an engine and the inference backends the application chooses. The feature matrix separates implemented paths from work that is not yet release-ready.
Local backends and remote providers share handles, request states, cancellation, streaming, and structured generation.
Unity, Unreal, and native callers poll bounded streams and marshal results on their own threads. Runtime hot paths do not invoke callbacks.
Agents retain history, summaries, prompt-cache state, and retrieved context even when their completed decode slot is released and reused.
Fixed-capacity queues, reusable scratch, and allocation gates protect maintained paths. Exact search remains the correctness oracle for compact vector lanes.
A prompt through Astral follows one request across the C ABI, durable agent state, continuous batching, provider execution, sampling, and engine-paced delivery. It separates validated behavior from paths that still need platform evidence.
The engineering notes keep the measurements and failed approaches alongside the implementation that survived them. The system tour starts with a prompt crossing the runtime.
recall / graph index
A graph index that looked finished was returning wildly different result quality from one query to the next, and more search budget only hid it. The write-up follows the diagnosis from symptom to a ground-up rewrite, including three competing fixes that lost on measurement first.
read the articlesimd / e5m2
x86 has no instruction that loads eight 8-bit floats and dots them, so five conversion paths competed for the kernel over one day. The write-up keeps the losers in the story, because two of them still serve machines the winner cannot.
read the articleconcurrency / mpsc
A queue's own retry counter reported a clean run while the cost curve was shaped exactly like contention. The write-up is about which instrument to believe when two of your own measurements disagree, and about the redesign that removed the contended branch entirely.
read the article