How coniti is verified

coniti 0.1.0 is a release candidate: built, tested and inspected, but not published. This page says what “tested” means here, so you can judge how far to trust it and what is left unverified.

The quality gate

One command, make check, runs everything below and must be green before any change is accepted:

Step	What it checks
Lint and format	ruff check and format check over the whole project
Types	mypy in strict mode over all 36 source files
Unit and end-to-end tests	423 tests: one module at a time, plus the checkpoint scenarios below
Chaos tests	432 tests that try to break things (next section)
Installed-wheel check	Builds the wheel, installs it into a fresh environment, and uses it from outside the source tree (see below)

In total there are 855 tests. Mutation checks are slower (about 30 minutes) and run separately, not as part of the gate.

Chaos testing

“Chaos” tests do not check one expected answer. They check that rules hold across many generated or hostile inputs.

• Property-based tests (Hypothesis) state a rule and let the library search for a counterexample. Example: the segment_tree structure must agree with brute force over the stored notes for any range, including at the edges.
• Oracle comparison. A simple, obviously correct model (a list, a brute-force scan) is run next to the real implementation and the two must give the same answer.
• Stateful testing. A state machine and seeded random walks of 500 steps drive the engine through ingest, ask, remove, snapshot and load, dump and load of a file (plain and gzip) and rebuild, comparing against a plain dictionary after every step.
• Adversarial sweep. Eight combinations of built-in embedders and structures face hostile texts, timestamps, ids, metadata and queries. Only the documented exception types may escape, every answer must serialise and render, and no NaN or infinity may appear anywhere.
• Fault injection. A memory backend that fails at random (5 to 40 per cent of operations) checks that memory and the structures still agree after every call. This is what found a real ordering bug in ingest, now fixed (see engine).
• Conformance suite for memory backends. Every backend must pass the same checks; the suite is itself tested against six deliberately broken backends (memory-backends).

End-to-end checkpoints

At each integration point several independently built parts were put together and run as a user would use them: configuration to engine to answers, memory moved by file and by snapshot with identical answers, a structure swapped in by configuration only, and a deliberate fault in the middle of an ingest. These checkpoints found problems that no single part’s tests could see, such as two engines silently sharing one memory.

The installed-wheel session

An editable install can pass while the real package is broken (a module left out of the wheel, say). So the gate builds the wheel and installs it into a throwaway conda-forge environment. From a directory outside the source tree, with a clean PATH, it then:

1. runs coniti --init and coniti.hello() and confirms the import comes from the new environment;
2. confirms the built-in kinds are discovered through the wheel’s coniti.plugins entry point;
3. runs a scripted session using only the public API: ingest notes, ask by text and by time, remove a note, write a gzip dump, and load_file it into a new engine.

Mutation checks

Passing tests show that the tests agree with the code. They do not show that the tests would notice a bug. Mutation checking tests that directly: the script changes one exact snippet of the source (for example removes a validation, flips a comparison, drops a rollback), runs the whole suite, and expects it to fail. The original file is always restored and verified byte-for-byte identical.

• 143 mutants across every module.
• 139 killed (the suite failed, as it should), 0 survived.
• 4 equivalent: changes that cannot alter behaviour, each with its reason recorded in the script (for example a value that is re-normalised afterwards).
• The first pass exposed gaps; 14 focused tests were added for them, then the whole run was repeated.

Structural rules

Two design rules are enforced by tests rather than by convention:

• No framework module imports coniti.impl. Implementations reach the framework only through the coniti.plugins entry point or a class path.
• No experiment parameter (polarity, threshold, decay, temperature) appears on a base class, a base spec or a framework contract model. Each plug-in declares its own.

The public API (__all__) is also checked to stay in sync with what the package loads.

What is not verified

• The dspy embedder was tested only against a fake dspy module, never against a real installation or model.
• Two behaviours are provisional decisions that may change: mixing embeddings from different spaces raises IncompatibleSpaces, and | and & on dense vectors have one built meaning among several possible (embedding).
• A failed rebuild after remove or load_file needs a manual rebuild(), and only the base vector is stored, so rebuilt structures do not see extended embedding fields (engine).
• The package has not been published or exercised by outside users.