Models and runtimes

Two inference runtimes behind one API, three execution providers, and a native dependency story that took a CI workflow to solve. This page is the mechanical half of the AI layer: what runs the models, and what that costs you in bytes.

Two runtimes, chosen per modality

There is no single local inference runtime that covers every model worth running, so the layer carries both. ONNX Runtime handles graph models with hardware execution providers. GGML — through llama.cpp and the GGUF format — handles quantised language and audio models, where the quantisation ecosystem lives.

Text to speech is the interesting row: it carries both backends behind one interface, because the good voice models are split across both formats. That is the clearest single illustration of what "one API over two runtimes" is buying — the calling code does not change when the model format does.

Execution providers

DirectML is the reason the free tier is not a token gesture: it is GPU accelerated on essentially any modern Windows machine, including integrated graphics. CUDA is faster on the hardware that supports it, and useless on the hardware that does not — which is why it sits behind the paid tier rather than being presented as the way to run models.

The native dependency problem

This is the unglamorous work that the layer actually exists to do. Running ONNX Runtime with CUDA on Windows means shipping a set of NVIDIA runtime DLLs whose versions must agree with each other and with the ONNX Runtime build that loads them. Get one wrong and the failure is a DllNotFoundException at generation time, on the user's machine, after they have already installed your application.

The repository solves it with a CI workflow rather than a documentation page. .github/workflows/package-cuda-cudnn-runtime.yml installs a pinned CUDA Toolkit, drops a pinned cuDNN redist into it, and packages exactly the runtime DLLs the application needs:

cuda_version:  12.8.1
cudnn_version: 9.8.0.87

sub-packages:
  cudart  cublas  cufft
  curand  cusparse
  cusolver  nvrtc

cudart64_*.dll     cudnn64_*.dll
cublas64_*.dll     cudnn_ops64_*.dll
cublasLt64_*.dll   cudnn_cnn64_*.dll
cufft64_*.dll      cudnn_adv64_*.dll
curand64_*.dll     cudnn_graph64_*.dll
cusolver64_*.dll   cudnn_heuristic64_*.dll
cusparse64_*.dll   cudnn_engines_*64_*.dll
nvrtc64_*.dll
nvJitLink_*.dll

The output lands as runtimes/win-x64/native/, alongside a generated runtime-manifest.json recording every DLL's file version and the exact path it came from. The workflow refuses to complete if either DLL set comes back empty, and it only ever copies from the resolved CUDA_PATH\bin so a preinstalled toolkit on the runner cannot contaminate the bundle.

One more collision worth knowing about

The Windows App SDK ships its own onnxruntime.dll. If your application is packaged as MSIX and also wants a CUDA-enabled ONNX Runtime, the SDK's copy wins by default and CUDA quietly never engages. The build fixes it by copying the self-built runtime over the top, after the normal output copy has finished:

<Target Name="ReplaceWindowsAppSdkOnnxRuntime"
        AfterTargets="CopyFilesToOutputDirectory"
        Condition="'$(IsWindows)' == 'true'">
  <ItemGroup>
    <_SelfBuiltOrtDlls Include="..\..\..\cudacudnn-runtime\win-x64\native\*.dll" />
  </ItemGroup>
  <Copy SourceFiles="@(_SelfBuiltOrtDlls)" DestinationFolder="$(OutDir)"
        OverwriteReadOnlyFiles="true" SkipUnchangedFiles="false" />
</Target>

SkipUnchangedFiles="false" is not decoration — the SDK's DLL and the self-built one can have the same timestamp and size, and MSBuild's default up-to-date check would leave the wrong one in place. This target lives in Apps/Engine/Engine.csproj, which means the runtime bundle has to be sitting beside the repository at ../cudacudnn-runtime/ for a Windows build to pick it up.

Model weights are never bundled

No model file ships inside the application, in any tier. Weights are large, licensed individually, and updated on their own schedule — embedding them would make every application update a multi-gigabyte download and quietly make you the redistributor of somebody else's license.

Manual acquisition is a real friction point, not a design principle, and an in-app model manager is an obvious thing to want. It is genuinely absent, and stating that plainly is better than the alternative of letting you discover it after installing.

Platform reach

The AI layer is far narrower than the engine, and it is worth being blunt about that before you plan around it.

The web row is not a limitation being worked around — it is verifiable. The WASM application's project file has exactly one project reference, and it is the engine. There is no code path by which AI could reach the browser build.