Runtime, Runnable and State

Runtime

tract.runtime.runtime_for_name(name: str)

Look up a runtime by name and return a Runtime instance.

Available runtimes depend on the build and the platform: "default" for CPU, "metal" on Apple Silicon Macs, "cuda" on systems with NVIDIA GPUs.

class tract.runtime.Runtime(ptr)

A hardware/software backend that can execute a Model.

Calling Model.into_runnable() implicitly uses the default CPU runtime. To run on a GPU, obtain a Runtime via runtime_for_name() — "metal" on Apple Silicon Macs, "cuda" on NVIDIA systems — then call prepare() to produce a Runnable optimized for that backend.

name() → str

Return the name of this Runtime.

prepare(model: Model) → Runnable

Prepare a model for execution on the Runtime.

NB: The passed model is invalidated by this call.

Runnable

class tract.runnable.Runnable(ptr)

A model that has been fully optimized and is ready to perform computation.

This is the final stage of the model pipeline. A Runnable is obtained either by calling Model.into_runnable() (CPU default) or by passing a Model to Runtime.prepare() for GPU-accelerated execution. Once obtained, call run() with numpy arrays or Tensor instances to perform inference.

input_count() → int

Return the number of inputs of the underlying model

output_count() → int

Return the number of outputs of the underlying model

input_fact(input_id: int) → Fact

Return the fact of the input_id-th input

output_fact(output_id: int) → Fact

Return the fact of the output_id-th output

property_keys() → List[str]

Query the list of properties names of the runnable model.

property(name: str) → Tensor

Query a property by name

run(inputs: List[Tensor | ndarray]) → List[Tensor]

Runs the model over the provided input list, and returns the model outputs.

spawn_state() → State

Create a new execution state for stateful (e.g. streaming) models.

profile_json(inputs: None | List[Tensor | ndarray]) → str

Profile the model. Also compute the static costs of operators.

Returns is a json buffer.

input_facts() → List[Fact]

Return the list of input facts.

output_facts() → List[Fact]

Return the list of output facts.

State

class tract.state.State(ptr)

Mutable execution state for stateful (typically streaming) models.

Stateful models maintain internal buffers between calls to run() (e.g. recurrent networks, pulsed convolutions). A State is created by Runnable.spawn_state() and can be called repeatedly with successive input chunks. Use freeze() to snapshot the state for later reuse.

input_count() → int

Return the number of inputs of the underlying model

output_count() → int

Return the number of outputs of the underlying model

run(inputs: List[Tensor | ndarray]) → List[Tensor]

Runs the model over the provided input list, and returns the model outputs.

freeze() → FrozenState

Freeze the state into an immutable FrozenState snapshot.

class tract.state.FrozenState(ptr)

A frozen (immutable) snapshot of a stateful model’s state.

Can be unfrozen back into a mutable State with unfreeze().

unfreeze() → State

Restore a mutable State from this frozen snapshot.