QIR

The QIR ("Query IR") is the shape a Rad client sends over the wire to describe a read, plus the sibling command IR for writes. This page specifies the grammar, how it lowers through the engine, the physical key-value layout it targets, and the exact KV operations each construct issues.

Everything here is traceable to code, arranged by the engine's downward-import stack: 06_frontend (public API) → 05_exec (execution, KV ops, codecs) → 04_planner (QIR → plan, access-path selection) → 03_lir (the IR and typed values) → 02_catalog (schema model + persistence) → 01_kv (ordered KV + order-preserving key encoding, over SlateDB). Imports only ever point downward; the wire ↔ IR lowering lives in cmd/rad/wireconv.go.

The pipeline

A read travels top to bottom; results are reassembled bottom to top.

The load-bearing idea, repeated throughout: the access path only narrows which keys are scanned; the filter remains the source of truth and is re-evaluated on every fetched row. Never assume a row was excluded because the access path skipped it.

Data and type model

Rad has exactly four scalar types.

Internal type	schema.rad keyword	JSON on the wire
text	string	string
int64	int64	JSON number (full 64-bit precision)
float64	float64	JSON number
bool	bool	boolean

Authors write type: string in schema.rad, but the internal catalog.Type — what /tables introspection reports and what this page uses below — is text (schema.go maps string → TypeText at parse). The other three match.

Any column may be nullable. NULL is a first-class value everywhere and sorts before every non-null value. format (uuid, unix_ms, email) is semantic metadata only; the engine never interprets it.

The runtime datum is lir.Value (03_lir/value.go): a tagged union over the four types plus a Null flag, with Row = map[string]Value keyed by column name. Two operators define all semantics: Equal (NULL never equals anything, including NULL) and Compare returning -1 | 0 | 1 (NULL sorts before every value; false < true; comparing two different non-null types is an error that a well-typed filter cannot produce, because literals are coerced to the column's type at lowering time).

The wire grammar

All reads — point lookups, lists, joins, aggregations — are a single Read sent to POST /query. There is no separate get or aggregate endpoint; a point read by primary key is just a Read filtered to the key columns with limit 1 (exactly what the generated Get / By<Unique> helpers build).

type Read struct {
    Table   string    // required
    Filter  *Expr     // optional predicate
    OrderBy []Order   // ordering terms, in list order
    Offset  int       // rows to skip
    Limit   int       // max rows; 0 = unlimited
    Include []Include // related relations to embed
    Aggs    []Agg     // present => fold matching rows to one scalar record
}

Filter — the Expr AST

Expr is a tagged union selected by op:

op ∈ { and, or, not, eq, ne, lt, lte, gt, gte, is_null }

and / or : { "op": "and", "exprs": [ <Expr>, ... ] }
not      : { "op": "not", "expr": <Expr> }
eq..gte  : { "op": "eq",  "column": "status", "value": "todo" }
is_null  : { "op": "is_null", "column": "assignee_id" }

• Comparisons test column OP value. Any comparison where either side is NULL is false — SQL three-valued logic collapsed to two-valued at the boundary. [POC]
• is_null is the only way to match NULLs.
• value is coerced to the column's declared type at lowering; a wrong-type value is rejected with an invalid problem, never guessed.

Ordering, includes, aggregates

Order is { column, desc }; ascending by default, with NULLs first ascending, last descending.

Include embeds a relationship: dir: "parent" follows an FK on this relation to the one row it references (→ nested object, or null), and takes no refinements; dir: "children" follows an FK on another table back to the rows referencing this one (→ array), and may be refined with filter / order_by / limit / nested include.

Agg is { fn, column, as } with fn ∈ count, sum, avg, min, max. Aggregation is a shape annotation on a relation, not a node in the Expr AST — the same slot that yields records yields a scalar fold when aggs is present. At the root it is mutually exclusive with order_by / offset / limit / include.

Fn	Result type	Empty input	Notes
count	int64 (never NULL)	0	count() counts rows; count(col) counts non-NULL
sum	column's numeric type	NULL	numeric only; NULLs skipped
avg	float64 (always)	NULL	numeric only; sum / count over non-NULL
min / max	column's type	NULL	any type; text lexicographic; false < true; NULLs skipped

Lowering: wire → IR

03_lir holds two IR shapes. The shaped read (lir.Read) is what the wire lowers onto: declarative, naming a table, filter, ordering, pagination, includes, aggs — and no access path. A parallel relational algebra (lir.Query over Scan / IndexScan / Filter / Project / Join / Limit) is the composable form reserved for future frontends (a SQL/GraphQL compiler would emit it); the wire path does not produce it today. This page follows the shaped-read path.

wireconv.toRead performs the lowering. The essential rule: every value is coerced to its column's declared type against the catalog, never guessed from JSON. It also introduces a spec-critical asymmetry in the Expr mapping:

• eq → lir.Eq{ ColRef, Literal }
• ne lt lte gt gte → lir.Cmp{ Op, ColRef, Literal }
• is_null → lir.IsNull{ ColRef }; and/or/not → the matching nodes

Equality is its own node; every inequality is a Cmp. As the access-path section shows, only lir.Eq drives index selection — so this split decides which filters can use an index.

Physical storage

Rad stores everything — catalog and data — in one ordered KV keyspace over SlateDB. One instance is exactly one database.

The KV abstraction

Four operations, and no more (01_kv/kv.go):

Get(ctx, key)            -> (value, found, error)
Put(ctx, key, value)     -> error             // overwrites; no separate Set
Delete(ctx, key)         -> error             // deleting a missing key is fine
Scan(ctx, start, end)    -> (Iterator, error) // half-open [start, end), ascending

• Scans are half-open [start, end), ascending lexicographic. nil start = from the beginning; nil end = to the end. There is no prefix primitive — a prefix scan is Scan(prefix, PrefixEnd(prefix)).
• No batch API and no standalone snapshot handle. The transaction is the snapshot and atomicity unit.
• Iterator keys/values are valid only until the next Next(); retainers must clone.

The load-bearing invariant: implementations must preserve lexicographic byte ordering of keys so range scans return rows in tuple order. SlateDB satisfies it directly.

Order-preserving key encoding

Every value encodes to [tag byte][payload]. Tags order mixed types; within a type the bytes sort in value order; every encoding is self-delimiting so tuples concatenate with no separator.

Type	Tag	Payload	Total
NULL	0x01	—	1 byte
bool	0x02	0x00 false / 0x01 true	2 bytes
int64	0x03	big-endian of uint64(i) XOR 0x8000_0000_0000_0000	9 bytes
float64	0x04	big-endian IEEE-754 bits, transformed (below)	9 bytes
text	0x05	body (each 0x00 → 0x00 0xFF), then terminator 0x00 0x01	variable

int64 flips the sign bit so big-endian bytes sort numerically; float64 sets the sign bit for non-negatives and inverts all bits for negatives, making the order monotonic across zero (NaN is rejected and can never appear in a key); text escapes embedded 0x00 and terminates with 0x00 0x01 so that, since 0x01 < 0xFF, a string always sorts before any extension of itself ("app" < "apple") and no encoded string is a byte-prefix of another.

int64   MinInt64 -> 03 00 00 00 00 00 00 00 00
        0        -> 03 80 00 00 00 00 00 00 00
        1        -> 03 80 00 00 00 00 00 00 01
float64 -1.0     -> 04 40 0F FF FF FF FF FF FF
        1.0      -> 04 BF F0 00 00 00 00 00 00
text    "eu"     -> 05 65 75 00 01
        "a\0"    -> 05 61 00 FF 00 01

PrefixEnd(prefix) returns the exclusive upper bound for a prefix scan: increment the last non-0xFF byte and truncate after it; return nil ("unbounded") if the prefix is empty or all 0xFF.

The keyspace

Top-level namespaces are literal ASCII path strings; only the tuple segments are order-preserving encoded bytes.

/rad/catalog/meta/next_id                 -> decimal-ASCII monotonic id counter
/rad/catalog/table/{table_id}             -> JSON of the Table struct
/rad/catalog/table_name/{table_name}      -> "{table_id}"   (name -> id lookup)

/rad/data/{table_id}/primary/{pk_tuple}   -> row value (column-id-keyed JSON)
/rad/index/{table_id}/{index_id}/{indexed_tuple}{pk_tuple}
                                          -> value = {pk_tuple}

• Physical identity is the table id (t1), never the name. Names live only in catalog metadata, so renaming a table rewrites one key and zero rows.
• The primary-key tuple is the PK columns encoded in order, concatenated with no separator; composite PKs are just more segments.
• An index entry is indexed_tuple ++ pk_tuple in the key, and the value is the PK tuple — the index is a covering pointer back to the base row.

Unique and non-unique indexes are physically identical (the PK suffix always makes entries distinct). Uniqueness is enforced at write time by prefix-scanning /rad/index/{table}/{index}/{indexed_tuple} and rejecting a different PK; under a serializable transaction the scanned range is tracked even when empty, so two concurrent inserts of the same value conflict at commit. [POC] NULLs participate in uniqueness as ordinary values.

Rows are column-id-keyed JSON

The value at a data key is JSON mapping column ID → lir.Value, not a tuple and not keyed by name:

{ "c2": {"type":"int64","int64":1},
  "c3": {"type":"text","text":"Al"},
  "c4": {"type":"text","null":true} }

MarshalRow translates name → ID on write; UnmarshalRow translates ID → name against the current table definition on read. Column values keyed by stable ID is the other half of rename-safety: a column rename rewrites catalog metadata and zero rows. A single monotonic counter issues every ID (t/c/i/fk prefixes), and IDs are never reused, so: adding a column makes existing rows read its literal default (never a generator like uuid()) or NULL; dropping a column orphans its field (ignored on read); re-adding gets a fresh ID so old data never resurrects.

Access-path selection

PlanRead turns lir.Read into a ShapedRead carrying the resolved table, the chosen Access, the full residual filter, ordering/pagination, includes, and aggs. There is no Sort/Filter/Limit/Aggregate plan node — those are fields the executor interprets; only the access path is a resolved physical choice.

chooseAccess reads exactly one signal: equalities(filter), which walks only the top-level AND-tree and collects column = non-null-literal predicates where the column is a bare ColRef on the left. Consequences:

• An or, not, is_null, or any Cmp node stops the walk and contributes nothing.
• Because ne/lt/lte/gt/gte lower to lir.Cmp (not lir.Eq), no inequality or range predicate ever selects an index — ranges survive only as residual filter. [POC]
• Reversed literal = column and NULL literals are invisible to selection.

Precedence: a complete-PK equality set → PK lookup; else the index with the longest gapless leading equality prefix → index scan; else full scan. The residual filter is the entire original predicate — even the equality that chose the index is re-checked on every fetched row.

Read execution

Engine.Read (committed) / Tx.Read (snapshot) → runShapedRead: fetch candidate rows via the access path, then either fold (aggregate) or filter → sort → offset → limit and attach includes.

Access	KV operations	Key range
pk_lookup	one Get	DataKey(table_id, pk_tuple) — 0 or 1 row
full_scan	one Scan, iterate all	[DataPrefix, PrefixEnd) — every row in PK order
index_scan	one Scan + one Get per entry	scan the index prefix, collect PK tuples, Get each

An index scan is therefore one Scan then a Get per matching entry:

A missing base row for a live index entry is a hard error — an integrity invariant worth exercising. Everything materialises: fetchRows returns a full row slice, and filter / sort / offset / limit run in Go memory. There is no predicate or limit pushdown into KV; the access path only narrows the scanned key range. [POC]

Filtering (evalRead) walks the Expr tree over decoded rows. The NULL rule, collapsed to two-valued logic [POC]: any comparison with a NULL operand is false, not of a false-because-NULL is true, and is_null is the only predicate true for a NULL. Ordering is a stable sort by each term (negated for desc, NULLs first ascending / last descending); then offset reslices; then limit truncates — all after the full matching set is materialised.

Relationships

There is no join operator. Nesting is a recursive, per-parent-row relationship fetch — a classic N+1 traversal.

A parent include maps the row's FK columns to the parent's PK: if any FK column is NULL the parent is null with no KV op, otherwise one Get, then recurse. A children include builds a want key from the FK columns and either index-scans (when the planner found a child index whose leading columns equal the FK columns) — one Scan + one Get per child — or [POC] full-scans the entire child table per parent row and filters in memory. Children refinements (filter / order_by / limit, offset always 0) then apply, and it recurses. Every parent row drives its own fetches; there is no batching across siblings.

Aggregation

foldAggs is a single fold over the already-fetched-and-filtered rows, one pass per term (the planner has pre-validated every term). A root aggregate chooses its access path from the filter exactly as a record read would (a count filtered by PK still rides a PK lookup), then folds all filtered rows into one Record of scalars; order_by / offset / limit / include are rejected at plan time. A children aggregate include folds the matched children into a single scalar object under its as name instead of an array, and may not also carry order_by / limit / nested include; parent aggregates are rejected outright. Typing and empty-set rules are in the grammar table above. [POC] the fold still materialises the rows it folds; overflow is not detected.

Mutations

Writes are the sibling command IR (/create, /update, /delete), each inside a serializable-snapshot transaction so the row write, its index entries, and its constraint checks commit atomically and race-safely.

Create applies defaults, normalises (reject unknown columns; absent nullable → explicit NULL; absent non-nullable → error; type-check), then: Get the row key (duplicate-PK check) · Get each non-null FK's parent (must exist) · Scan each unique index prefix (reject a different PK) · Put the row · Put an entry for every index.

Update loads by PK (absent → not found), rejects any PK column in the patch (the PK is immutable), merges and re-normalises (clear to NULL by passing a NULL value; defaults are not re-applied), re-checks only the constraints the patch touches, Puts the row at its stable key, and for each index whose columns changed, Deletes the old entry and Puts the new one.

Delete loads by PK, then FK-restrict: scans all tables for any FK referencing this row and errors if one exists ([POC] restrict, never cascade); then Deletes every index entry and the row.

Defaults (05_exec/defaults.go) apply at insert only: uuid() → crypto/rand v4 string, now_ms() → time.Now().UnixMilli(), or a typed literal. Explicit values win, including an explicit NULL — a present key is never overwritten by a default.

Migration, transactions, result shape

Migration diffs the current catalog against the desired schema and applies each step in order (add/drop table, add/drop column, rename via renamed_from, add/drop index); it is idempotent. Adding an index registers metadata then backfills entries for existing rows in one transaction, failing the whole migration if the data already violates a unique index. Adding a column rewrites no rows.

Transactions offer two isolation levels: Snapshot (stable reads + own writes; write-write conflicts only) and SerializableSnapshot (also read-write conflicts — point reads and the requested bounds of scans are validated, so a phantom insert into a scanned range conflicts even if the scan returned nothing). All mutations run serializable, which is what makes the constraint checks race-safe. Commit awaits durability and either applies atomically or returns an error wrapping kv.ErrConflict (retry with fresh reads, or abandon). Rollback is idempotent and a no-op after commit. The 01_kv/kvtest suite is the executable definition of these semantics.

Result shape — a read returns []*Record; the frontend renders each into one flat JSON object by merging four maps, dispatching by relation kind: Columns → scalar fields; Parents[as] → nested object (or JSON null); Children[as] → array of objects; Scalars[as] → nested scalar object (a folded aggregate include). A root aggregate is a single flat object. Rad never exposes a flattened join; output is always shaped like the request, with keys sorted for determinism.

A worked example, to the byte

Schema (IDs from the shared counter): table users (t1) with id (c2, int64, PK), name (c3, text), email (c4, text), and a unique index users_email_idx (i5) on email.

Insert id:1, name:"Al", email:"a@b.co". The PK tuple for id=1 is 03 80 00 00 00 00 00 00 01:

row:
  key /rad/data/t1/primary/ ++ 03 80 00 00 00 00 00 00 01
  val {"c2":{"type":"int64","int64":1},
       "c3":{"type":"text","text":"Al"},
       "c4":{"type":"text","text":"a@b.co"}}

index entry (email "a@b.co" -> 05 61 40 62 2E 63 6F 00 01):
  key /rad/index/t1/i5/ ++ 05 61 40 62 2E 63 6F 00 01 ++ 03 80 00 00 00 00 00 00 01
  val 03 80 00 00 00 00 00 00 01                          (the PK tuple)

A query filtered by email extracts eqs = {email}, misses the PK, matches the index's leading column → index scan: one Scan of the index prefix collects the PK tuple, one Get on the data key loads the row, and the full filter is re-checked before it is returned. A query filtered by id instead covers the PK → a single Get, no scan.

Conformance invariants

The properties a QIR test harness should pin down:

• Encoding: round-trip and order-preservation per type; cross-type tag order null < bool < int64 < float64 < text; string prefix-safety; PrefixEnd bounds.
• Access paths: the result set is identical whichever path is chosen (the filter is authoritative); PK-complete → lookup, longest gapless index prefix → index scan, else full scan; or/not/inequalities/is_null/reversed equality/NULL literals never select an index.
• Reads: three-valued logic (comparisons with NULL false; is_null the only match; not of false-because-NULL true); NULLs first ascending / last descending; offset/limit after sort; children offset always 0.
• Relationships: NULL parent FK → null, no KV op; no-index children via full-scan fallback; nested shapes reassemble correctly.
• Aggregation: per-fn typing and empty-set rules (count → 0, everyone else → NULL); NULL-skipping; the root/children/parent restrictions.
• Mutations: duplicate PK / missing FK parent / unique violation rejected; PK immutable; clear-to-NULL; only touched indexes maintained; FK-restrict on delete; every live index entry points at an existing row.
• Transactions: the whole of 01_kv/kvtest.

POC deviations and non-goals

Collected for a future spec to rule on: three-valued logic collapsed to two-valued at the filter boundary; NULLs count in unique constraints; only equality predicates drive access paths (no range index scans); no predicate/limit/offset pushdown (reads materialise the full matching set); includes are unbatched N+1 with a full-scan fallback; aggregates materialise before folding and overflow is undetected; deletes restrict, never cascade; a failed unique-index backfill leaves the index registered but empty.

Out of scope entirely: GROUP BY / HAVING / DISTINCT, window functions, recursive queries, CTEs, unions, cost-based optimisation, join reordering, an expression language beyond the above, and any SQL endpoint.