Livt.ML

Fixed-size machine-learning building blocks for Livt hardware-oriented designs.

Livt.ML provides small, explicit components for integer vectors, quantized linear inference, activation functions, embeddings, attention, normalization, transformer-style composition, and simple fitted models. The package is intentionally concrete rather than generic: APIs expose fixed sizes, stored outputs, and visible scale factors so generated hardware remains easy to inspect.

📦 Package

[dependencies]
"Livt.ML" = "1.6.1"

Version 1.6.1 publishes package discovery metadata, this README, the changelog, and the MIT license. Its component API is unchanged from 1.6.0.

Livt.ML 1.6.1 depends on Livt.Math 1.7.2 for reusable arithmetic and Livt.IO 1.8.1 for RAM-backed tensor storage. See CHANGELOG.md for release notes and compatibility guidance. Memory sources use the generic Livt.IO API: byte tensors use scheduled BlockRam<byte, 2048> banks, and causal attention uses AsynchronousDistributedRam32x32. Run livt sync after updating dependencies. Livt.Math 1.7.2 and Livt.IO 1.8.1 must be available in your registry or local workspace before resolving this version of Livt.ML.

📚 API Overview

Most stateful components follow the same call shape: configure any stored weights or tables, call Compute(...), then read results with GetOutput(...). For the simple fixed-size components, out-of-range getters return 0 and setters ignore the write. Checked matrix and attention APIs instead report failure explicitly and may invalidate retained results; follow their contracts. Optimized components ending in Stream use their documented toggle or ready/valid handshake and bounded internal storage; see docs/streaming.md.

ClassProjection, ScaledDotProductAttention, and TransformerBlock remain experimental compositions. Their APIs may evolve when exercised by an imported language model; the numeric, tensor, RAM, and streaming operator APIs form the current stable inference foundation.

using Livt.ML.Linear

component ExampleLayer
{
	layer: Linear8x8

	new()
	{
		this.layer = new Linear8x8()
	}

	public fn Run(input: int[8])
	{
		this.layer.SetRowValues(0, 1, 0, 0, 0, 0, 0, 0, 0)
		this.layer.Compute(input)
		var first: int = this.layer.GetOutput(0)
	}
}

🔢 Data Model

The API uses fixed-size int vectors, mostly int[8]. Signed-int8 tensor elements are represented by int values in [-128, 127]; quantized linear layers retain signed int32 accumulator results until callers explicitly invoke RequantizeInt8. Several components expose integer scale constants:

• SigmoidLUT, SiLUApprox, and SoftmaxApprox use scaled integer approximations.
• ScaledDotProductAttention uses explicit score and weight scale constants.
• LinearRegression stores the fitted slope as aFixed, scaled by LinearRegression.SCALE.
• DotProduct.ACC_MAX defines the positive saturation limit used by the linear projection components.
• LayerNorm64, GeluNewApproxQ8, and their learned affine values use Q8.8 scale 256; StableSoftmax8 emits Q1.15 weights at scale 32768.
• Int8LinearAccumulator retains signed int32 sums and deliberately does not choose output scales; model importers supply those after calibration.
• Int8BroadcastMac64x8 and Int8Dot64x8 accept signed 16-bit activations, packed signed-int8 weights, and retain signed int32 accumulator results. Callers must prove that activations fit [-32768, 32767] before the boundary.

Approximation components are useful for simulation, reference designs, and early hardware exploration. Timing-critical designs should review docs/hardware-notes.md.

🧪 Build and Test

Build the package:

livt build

Run the configured test components:

livt test

The test list is defined in livt.toml. Short call-order examples live in docs/usage.md.

🔧 Layout

src/numeric/      ML-specific quantization and requantization
src/linear/       vector addition, dot products, and dense projections
src/pooling/      tensor-reference and line-buffered streaming pooling
src/storage/      banked byte tensors backed by Livt.IO RAM components
src/activation/   ReLU, sigmoid/SiLU/GELU approximations, and stable softmax
src/norm/         RMSNorm and reference/streaming LayerNorm
src/embedding/    fixed tables and external-memory embedding streams
src/attention/    KV cache plus reference and bounded causal attention
src/classifier/   small classifier helpers
src/transformer/  fixed-size transformer-style blocks
src/model/        small fitted model components
tests/<domain>/   tests mirroring each source domain
docs/             usage examples and package notes

Source namespaces mirror their folders, for example Livt.ML.Attention. Test namespaces mirror the same structure below Livt.ML.Tests.

📖 Notes

• Usage examples
• Numeric model
• Package boundaries
• Hardware notes
• Streaming inference
• Checked INT8 matrix accumulation
• Checked block attention

Compiler bugs should be recorded only in COMPILER.md. Hardware tradeoffs and implementation notes belong in docs/.

🚀 Outlook

Livt.ML provides a broad set of fixed-size components. Future work should add more hardware-optimized variants, shared fixed-point helpers through Livt.Math, and stronger model-weight loading patterns.

📄 License

This project is licensed under the MIT License. See LICENSE.

Generic wide fixed-point operators

Version 1.3.0 adds streamed INT4 projection, fixed-point RMSNorm, masked softmax and lookup activation with explicit formats and injected storage. See contracts. Existing operator semantics are unchanged.

Generic fixed-point transformer operations (1.4.0)

ITransformerKernels and FixedTransformerKernels provide serialized embedding, affine RMS normalization, matrix projection, unscaled masked attention with relative position bias, table-driven activation/gating and residual addition. Token/vector bounds, fixed-point formats, epsilon, external IRam<int> storage, weights and activation tables are generic parameters. One worker can serve every layer. This is a serial correctness implementation, not a throughput or FPGA resource guarantee.

Attention scratch is an explicit generic IRam<int> dependency with at least 2 * MAX_TOKENS words. See the source contracts in ITransformerKernels and FixedTransformerKernels for ownership, reset, capacity, scheduling and completion requirements.

QuantizedTensorWeights selects immutable ITensorCatalog records and reads low-nibble-first packed INT4 codes and unsigned scale words from external memory. Addresses count int32 words except catalog packed-code addresses, which count bytes. It handles unaligned byte offsets and flattened groups crossing matrix rows. Invalid selection/reads latch a failure until a successful selection.

A kernel returns false on invalid dimensions, ranges, values or provider failure; its destination may then contain partial writes and must be discarded. Input and model spans must remain stable for the call. The memory provider must expose contiguous valid ranges. Only residual addition permits exact input/output aliasing; partial overlap is rejected. The caller owns serialization and result validity. FixedTransformerKernelsTest covers a different Q8 format and shape from FLAN-T5, including the deliberate lack of probability-sum correction.

Incremental transformer operations (1.4.1)

IIncrementalTransformerKernels extends the existing encoder interface with single-query CachedAttention and ProjectArgMax; existing implementations of ITransformerKernels need no changes. FixedTransformerKernels implements both through interface inheritance and reuses its arithmetic and provider instances.

Cached attention takes an independent K/V length, query position, causal flag, key mask and optional bias tensor. A bias ID of -1 disables bias. Bucket entries are int32 words covering relative distances -(MAX_TOKENS-1)..MAX_TOKENS-1. Masked/future K/V entries are not read. Scratch scores for excluded keys are unspecified; their probabilities and contribution to context are zero.

Streaming projection supports up to 65,536 output rows independently of the maximum vector width. It rounds and clips every complete dot product before feeding StreamingArgMax, preserves lowest-index ties, and returns -1 on error. It allocates no output-logit tensor. Tests cover a full 32,128-row scan, clipping that creates a tie, poisoned masked/future cache entries, Q8 and existing kernel contracts. Model cache addresses and lifecycle policy stay in application code.

Scheduled integer division

Division policy and migration status describe the shared Livt.Math quotient/remainder API and its use in transformer shape checks. Runtime division is explicitly scheduled; compile-time arithmetic and existing fixed-point rounding retain their own contracts.

Checked INT8 matrix kernel (1.6.0)

Linear.Int8MatrixAccumulator32<ACTIVATION_BITS, PRODUCT_STAGES> adds signed int32 activations, INT8 weights and checked int64 output sums retained inside the engine. Four packed activation loads configure a 64-column chunk; live weight rows then update up to 1,024 output sums without returning partial dots to the caller. Product stages 1 and 2 preserve identical arithmetic.

The caller owns memory transport, row layout, final scaling and rounding. Existing 16-bit kernels remain unchanged. See the matrix contract for packing, bias, overflow, ownership, reset and validation details.

livt test --run Int8MatrixAccumulator32Test

Checked block attention

Attention.AttentionAccumulator32<PRODUCT_STAGES> retains a 32-element query and computes exact Q.K scores and weighted-V numerators through Livt.Math.Arithmetic.CheckedBlockMac32. Each vector is supplied as two packed 512-bit blocks of sixteen signed int32 lanes. Results contain a checked int64 sum and a failure flag. Scaling, softmax and final division stay with the caller.

AttentionAccumulator32Port<PRODUCT_STAGES> accepts a live 512-bit values input and schedules control operations, avoiding full-row argument copies. Hold each row stable until the method returns and submit blocks in order 0, 1. Invalid query/score order fails; invalid or overflowing weighted updates poison reads until clear/reset. Both variants require a single owner and serialized calls. PRODUCT_STAGES selects 1 (default) or 2 without changing arithmetic.

These additions preserve existing INT8/16-bit kernel APIs and defaults. See the attention contract for row lifetime, block ordering, overflow and reset behavior. Run the paired-row generated-wrapper regression with the Livt compiler/simulator:

livt test --run AttentionAccumulator32Test

Batched INT8 matrix interface (1.6.0)

Linear.Int8MatrixAccumulator32Stream<ACTIVATION_BITS, PRODUCT_STAGES> shares its checked core with the existing blocking interface and adds AccumulateRows(first, count). A ready/valid producer retains each packed 64-byte weight row until acceptance. Eight lanes feed continuously within a batch while completed rows update BRAM; model layout, memory transport and final scaling remain caller responsibilities. No DDR or board dependency is introduced. See the stream contract.

livt test --run Int8MatrixAccumulator32StreamTest

The producer must retain valid/data until the transfer edge; ready releases its row storage after eight feeding cycles. Batch completion includes all BRAM writes. Source failures and overflow invalidate the bank until Begin or reset. The native core uses synchronous active-high reset; reset both ends together across a rising edge and call Begin before reuse.