Engine¶

Built and verified

ContinuityEngine, Answer, AnswerHit, GroupView, EmbeddingMismatch and SnapshotError (defined in coniti.broker, exported from coniti) are built and tested end to end, including the lifecycle (remove, rebuild, snapshot, load, dump, load_file) and atomic ingest. Whole-engine stateful and adversarial testing and mutation checks have passed; see quality for how coniti is verified.

The ContinuityEngine wires the four parts together: memory, an embedding embedder, structure structures and a strategy. You give it notes (ingest) and questions (ask); there is no session object.

from coniti import (
    Answer, AnswerHit, ContinuityEngine, EmbeddingMismatch, GroupView,
    Note, SnapshotError,
)

They are part of the top-level public API; coniti.broker is where they are defined and also works.

Creating an engine¶

ContinuityEngine(memory=None, embedder=None, structures=None, strategy=None,
                 *, namespace=(), container=None)
ContinuityEngine.from_config(source)    # a path to a config file, or a mapping

Where the parts come from. Each part is taken from a container:

the container argument, if you pass one;
otherwise the innermost container made active with use(...);
otherwise a fresh Container.

The fresh container matters: two engines created without a container never share memory. (An earlier version fell back to a process-wide default container, so two “default” engines silently shared one memory. This was found in review and changed.) A fresh container still reads ./coniti.toml and CONITI_CONFIG like any other (see configuration), so an engine created in a directory with a coniti.toml uses it.

Explicit parts win. Each argument may be a ready instance, a spec, or a mapping such as {"kind": "local_hash", "params": {"dim": 32}}; structures is a list of those. Each explicit part is applied as a temporary override of the container while the parts are built, so the parts that depend on it follow it:

e = ContinuityEngine(embedder={"kind": "local_hash", "params": {"dim": 32}})
e.embedder.dim                       # 32
[s.dim for s in e.structures]        # [32]   the default flat structure got 32 too

If you pass structures explicitly with the wrong dimension, the engine raises a ValueError (“FlatStructure has dim 16 but the embedder produces 32”).

Read-only properties: memory, embedder, structures (a copy of the list), strategy, namespace, container, context (the EngineContext the strategy sees) and specs (the spec ids of the parts, by piece).

namespace addresses a slice of memory (see memory); it is not a session.

Ingest¶

Method	Behaviour
`ingest(note) -> bool`	Embed the note, ask the strategy to admit it, insert it into every structure, then write it to memory (last). Returns whether it was admitted. Atomic (below).
`ingest_many(notes) -> int`	Ingest each note in order; returns how many were admitted. Atomic per note; stops at the first failure (below).

The embedding id is written to the memory’s meta on the first write. A structure receives the full embedding output (so a plug-in structure can use extra fields, such as the source_vector of a projected embedding).

The engine keeps no copy of notes or vectors. Everything it shows comes from memory, and structures are derived data. Consequences:

A new engine over memory that already holds records rebuilds its structures from that memory and answers at once. Only the base vector is stored, so a rebuilt structure gets no extended embedding fields.
Memory written with a different embedding is refused: the engine raises EmbeddingMismatch (a ValueError) at construction, naming both embedding ids. Use the same embedding, or another memory.

a = ContinuityEngine()                      # default embedding
a.ingest(Note(id="x", text="hello", timestamp=1.0))
ContinuityEngine(memory=a.memory, embedder={"kind": "local_hash", "params": {"seed": 1}})
# EmbeddingMismatch: memory (namespace ()) holds embeddings from 'emb-74006f08273e'
#   but the engine's embedder is 'emb-0ef02b167165'; use the same embedding or another memory

Atomic ingest¶

ingest works by compensation, so it needs nothing from the memory backend beyond its normal operations. Memory is written last, so a rollback never has to change memory records. The order is:

embed the text;
ask the strategy to admit the note (a rejected note changes nothing);
refuse an id that memory already holds (DuplicateRecord);
insert the record into every structure;
write the embedding id to the memory’s meta, only on the first write;
write the record to memory (put, all-or-nothing).

A failure in step 1, 2 or 3 changes nothing. A failure in step 4, 5 or 6 resets the meta this call wrote, rebuilds every structure from memory and re-raises the original error. Either way, memory, meta and every structure describe the same notes as before the call. A failed first ingest does not set the embedding id, so the memory can still be used with another embedder.

Because a failed ingest never writes a record, memory only ever holds records that every structure accepted. That is what makes rebuild() always succeed, and it can repair the structures even if a rollback step itself failed (for example if the meta reset or the rebuild inside the rollback raised).

Why memory is last

An earlier version wrote memory first and deleted the record on failure. The adversarial sweep (a memory backend that fails at random) found that when that compensating delete failed, a record the segment_tree had rejected stayed in memory, and rebuild() then raised forever. Writing memory last removes the problem: the rollback only touches derived data.

This was checked with a real fault: a segment_tree rejects a timestamp outside its range while the note is being inserted. The call raises the structure’s own ValueError, the note is never written to memory, and the other structures are intact:

e = ContinuityEngine(structures=[
    {"kind": "flat"},
    {"kind": "segment_tree", "params": {"bucket_seconds": 1000.0, "branching": 4, "levels": 6}},
])
e.ingest(Note(id="ok", text="fine", timestamp=10.0))
e.ingest(Note(id="late", text="too late", timestamp=1e9))
# ValueError: timestamp 1000000000.0 is outside the segment tree's range [0, 4096000.0)
#   (bucket_seconds=1000.0, branching=4, levels=6)
e.memory.count(e.memory.count_request()).count      # 1   only "ok" is stored

ingest_many is atomic per note, not as a batch: it stops at the first failure, re-raises it, and the notes before the failing one stay ingested.

Remove, rebuild, snapshot, load¶

Method	Behaviour
`remove(note_id) -> bool`	Delete the note from memory and rebuild the structures. Returns `False` (and changes nothing) if there was no such note.
`rebuild() -> None`	Reset every structure and insert what memory holds, in order. Safe to call at any time.
`snapshot() -> dict`	The engine’s namespace as plain data: `{"version": 1, "embedding_id": ..., "namespace": [...], "records": [...]}`.
`load(snapshot) -> int`	Load a snapshot into the engine’s memory; returns the number of records. All-or-nothing.
`dump(target) -> int`	Write the engine’s namespace to a memory file (a `.gz` name is gzipped).
`load_file(source) -> int`	Load a memory file into the engine’s namespace, then rebuild the structures.

A snapshot and a memory file hold the same thing. Each snapshot record has exactly the shape of one record line of a memory file (MemoryRecord.to_dict()), so snapshot()["records"] can be written as a file’s record lines and the other way round. Vectors are stored, so nothing is re-embedded. Only the base vector is kept (see the limits below).

load validates everything before it writes anything. It checks the version, the embedding id (EmbeddingMismatch), every record, each vector’s shape, duplicate ids inside the snapshot and ids that already exist in memory. Any problem raises SnapshotError (the embedding id problem raises EmbeddingMismatch) and writes nothing. Records are re-homed into the engine’s namespace. After the writes it rebuilds the structures and sets the embedding id in the meta last; if a write fails, the records it wrote are deleted again and the original error is re-raised. load and load_file record the embedding id even when the snapshot or file holds no records, so an empty load still ties the memory to this engine’s embedder.

load_file checks the file’s embedding id first. It reads only the header (coniti.memory.read_header), raises EmbeddingMismatch if the id differs from the engine’s embedder, and only then loads. A problem in the file raises MemoryFileError and writes nothing. Neither load nor load_file merges over existing records; to replace, clear the memory or load into a new engine. See backup-restore for examples.

Limits¶

If the structure rebuild fails after load_file has written, memory keeps the loaded data and the structures may be left reset. Call rebuild().
If the rebuild fails after a remove, the note is already gone from memory. Call rebuild().
Extended embedding fields (for example the source_vector of random_projection) are not kept in memory, so structures rebuilt by rebuild, remove, load or load_file see only the base vector. The structures fed live by ingest do see the extra fields.

Ask¶

ask(query) -> Answer embeds the query text (if any), calls the strategy, and turns the ids it returns into notes. A Query needs at least one of text, time_range or near (see layers).

Part of `Answer`	Meaning
`query`	The query asked.
`hits`	`AnswerHit(note, score)`: matching notes, best first. A hit whose id is no longer in memory is dropped.
`groups`	`GroupView(group, representative)`: one per group a structure returned.
`raw`	Each structure’s own output, keyed `"<index>:<ClassName>"`, for example `"1:SegmentTreeStructure"`.
`render()`	A readable summary (hits with scores; groups with count, span and representative).

Answer is a normal contract model: to_dict, to_json, from_json and Answer.json_schema() work, and a round trip gives an equal answer (see contracts).

Representatives¶

A group is a summary (a count, a centroid, a span). The representative is a real note that stands for it: the stored note inside the group’s time span whose vector is most similar to the group’s centroid. The similarity uses the embedding algebra (centroid @ vector). It is None when the group has no centroid or span, or a centroid of the wrong size (for example from a structure in another dimension).

One ask can combine both kinds of answer. With flat and segment_tree configured, a text query with a time range returns hits from flat and a group from segment_tree (see quickstart for real output).