Skip to content

Changelog

All notable changes to kglite-docs are recorded here. Format loosely follows Keep a Changelog; versions follow SemVer, but pre-1.0 minor releases may include breaking changes (called out below).

Unreleased

[0.0.15] — 2026-05-31

OCR legibility & portability, from a full-corpus run on 0.0.14 (745 pages OCR'd). Makes "OCR'd" mean "readable" (illegibility surfaced, not silently counted), lets you re-OCR + escalate bad pages, right-sizes/tiles the image so fine print survives, makes OCR portable (export/import sidecar) and OCR'd text first-class (embedded + searchable), pins model guidance (Sonnet default; small models fabricate), and adds a structured batch submit. Plus versioned study reports on the graph so analysis stops cluttering the folder. Also folds in the Read the Docs build fix (static docs, no runtime stack).

Added — versioned study reports (on the graph, not .md litter)

  • study("report", name=…, text=…) saves a markdown report on the study node — named (several distinct reports per study coexist) and append-only versioned (each save is a new version; full history, latest-wins on read). study("reports") lists names + versions; study("get_report") reads (latest or a specific version); study("export_report") writes one to disk on demand — so reports live in the .kgl (portable, diffable, queryable) and agents stop cluttering the working folder with one-off files. A report can record the cites (finding/assessment ids) it rests on; reports cascade-delete with the study.

Added — library-owned batch OCR submit

  • ocr("submit_many", rows=[{page_id, markdown}|{page_id, tiles}]) submits many pages in one call. rows is a structured tool argument (the SDK escapes it), so agents never hand-serialize multi-line, quote-heavy verbatim transcriptions into a JSON file — the failure mode that silently corrupted ~half a batch in the field run. A failing row is reported, not fatal.

Added — OCR model guidance

  • The request_ocr task now carries a recommended_model (claude-sonnet-4-6)
  • model_guidance, and _INSTRUCTIONS documents the tier: Sonnet default, Opus for the hardest/decisive pages, and avoid small models for legal/forensic OCR — A/B testing found Haiku corrupted names and fabricated an entire false document (contamination, not low quality). kglite-docs bundles no model; this is guidance only.

Added — OCR export/import sidecar + OCR'd chunks are first-class

  • ocr("export", doc_id) writes a document's OCR to a sidecar JSON (<source>.ocr.json) — {source_file, doc_id, generated_at, by_model, pages:[{page_number, ocr_model, ocr_status, legible_chars, text, …}]} — so OCR is portable, auditable, diffable, hand-correctable, and re-importable; ocr("import", path) round-trips it back in (done once; a human fixes the failed pages and re-imports; OCR travels with the PDF). Carries the illegibility flag.
  • Fixed: OCR'd chunks are now marked embedded. submit_ocr embedded the chunks but didn't set embedded/:Embedded, so search() counted them as unembedded — warning they were "invisible" (they weren't) and, for a scanned-only document, raising NotIndexedError despite the vectors existing. OCR'd text now lives on the graph exactly like native-text chunks: :Ready
  • :Embedded, searchable/studyable, with ocr_derived honesty markers.

Added — adaptive image prep + tiling (attack illegibility at the source)

  • The library right-sizes the page image it ships for OCR. A page rendered larger than a vision model's input (~1.15 MP) is silently downscaled by the API, destroying fine print on dense/faded scans (a chunk of the illegible 25%). Now request_ocr returns the page as tiles fit to that cap: one full-page tile if it fits, else detail-preserving overlapping tiles rendered at full dpi (render_page_images). submit_ocr accepts tiles=[{tile_index, markdown}] and stitches them in order (or whole-page markdown as before). Preview renders (list_pending_ocr/illegible) are downscaled under the cap too. The page records ocr_render_dpi/ocr_tiles for provenance ("illegible even after tiling" vs "illegible because downscaled"). All in Python via pymupdf — no Rust, no new deps.

Added — re-OCR an already-done page (force)

  • ocr("request", …, force=true) re-OCRs a page that's already transcribed — so an illegible/partial result can be escalated to a stronger model (ocr("illegible") → re-request with force → re-submit). submit_ocr now replaces all of a page's chunks (not just the needs_ocr placeholders), so a re-OCR overwrites cleanly instead of leaving duplicate chunks.

Fixed — silent illegibility: "OCR'd" ≠ "readable"

  • A page OCR'd to all-[ilegível]/empty no longer counts as covered. A full-corpus run found ~25% of OCR'd pages came back illegible/near-empty, yet ocr_status read "done" — the same silent-incompleteness failure as the original scanned-pages-pass-as-ready bug, one stage later. submit_ocr now computes a legibility metric (legible alnum after stripping […] markers) and records a page ocr_outcome (ocr_ok / ocr_partial / ocr_illegible) + legible_chars. ocr_status reports illegible_pages / partial_pages / readable_pages (the honest count); coverage_report says so loudly; and ocr("illegible") lists the unreadable pages as a retry worklist. The illegibility tag is computed by the library, not narrated by the agent.

[0.0.14] — 2026-05-30

Scanned primary evidence stops hiding. A field report found the strongest fact in a case — the opposing party's sworn confession on a scanned page — invisible out of the box: the page was mis-flagged ready, never queued, never OCR'd. This release makes scans detectable, makes OCR a lazy agent-driven step, and adds a source-party dimension so an admission against interest surfaces as the adverse party's own primary text. Agent-first throughout; bring-your-own OCR engines deferred.

Added — source-party dimension (whose words a document holds)

  • Tag a document with who produced/filed it, inherited to its chunks, so an admission against interest (primary text by the adverse party) surfaces instead of sinking as someone else's characterization of it. provenance says what was checked; source_party says whose words. Set it at ingest (document("ingest", …, source_party="respondent")) or after (document("set_party", doc_id=…, party=…)); it becomes a queryable label (MATCH (c:Chunk:Respondent)), shows on document("get"), and is surfaced per row in the study ledger (source_party) — so "primary-text supporting evidence by the adverse party" is one query. Generic + domain-opaque (free-text → PascalCase label); the legal pack registers claimant / respondent / defense / accuser / judge / court / expert / third_party for discovery (document("parties")). Document-level for now (per-chunk override is future).

Added — lazy, agent-driven OCR (ocr("request"))

  • The first time an agent needs a scanned page, it's handed the OCR task to do itself instead of getting empty text. ocr("request", page_id=…|doc_id=…, page_number=…, agent_id=…) returns the rendered page (image_b64) plus a strict verbatim transcription prompt (OCR_PROMPT — transcribe exactly, mark [illegible], no inference; it will be quoted as evidence); the agent transcribes and calls ocr("submit"). kglite-docs still ships no OCR engine — the vision-capable agent is the engine (agent-first). Pass agent_type to route the task to a specific OCR subagent (echoed + recorded; the library never dispatches it). The request is recorded on the page (who/when/agent_type, first request preserved). submit_ocr now stamps OCR'd chunks ocr_derived (+ ocr_by/ocr_model) so a reviewer knows to eyeball the image before quoting. Requesting a page that isn't needs_ocr raises.

Fixed — robust OCR-requirement detection

  • Scanned pages stop silently passing as ready. A field report found the single strongest fact in a case — the opposing party's sworn confession on a scanned page — invisible because the page was mis-flagged ready and never queued for OCR. Detection is now image-coverage- and density-aware (not presence + an absolute char floor): a page is flagged needs_ocr when it's image-bearing with sparse text or image-dominated without being text-rich — catching full-page scans whose raster isn't a detected XObject, and scans whose junk fragments clear the old 120-char floor. Recall-biased (over-flagging a figure page is cheap; missing an exhibit is not). PageContent now carries image_coverage.

[0.0.13] — 2026-05-30

Cross-chunk synthesis & leveled studies — from four 2026-05-30 field reports. The per-chunk assess/verify model was silently blind to emergent cross-chunk patterns (disparate treatment, conflicting dispositions); a study could be concluded "complete" while a whole class of finding was unreachable. This release makes that class first-class, observable, and gated — on generic primitives, with the legal vocabulary registered as a data pack.

Added — regression fixture + test helper

  • Cross-chunk regression fixture wired into CI. Adopted sample_data/cross_chunk_fixture/ (synthetic, anonymized case TR-7788) and a test that locks the whole arc end-to-end: the per-chunk baseline recovers nothing (study_conflicts == 0 — the trap), a synthesis pass recovers F1 (disparate treatment) + F2 (conflicting dispositions) + the omission and passes the fixture's own gold-standard check() while leaving the negative control alone, and conclude is refused until the study is synthesized.
  • kglite_docs.testing.make_chunks(corpus, n) — a public helper that deterministically produces n chunks (encoding the structure-aware lever short paragraphs otherwise pack into one), de-duping a helper the test suite re-pasted.

Added — timeline / Event layer

  • Queryable entity-value scalars on chunks. Ingest now stores date_first, date_count, money_max, money_sum, money_count as chunk properties (from the same regex entity pass), so timelines and aggregates work in Cypher (ORDER BY c.date_first, sum(c.money_max)) without re-parsing entities_json.
  • Event layer + sequence analysis. study("add_event", …) records a dated occurrence (date / actor / action / outcome, optionally chunk-anchored; study("event_prompt") is the extraction prompt); study("timeline") returns a document's events in order; study("timeline_conflicts") runs deterministic sequence analysis — disparate treatment (same trigger → different outcome by actor) and contradictory outcomes (same actor+action, conflicting outcomes) — completing the disparate-treatment detector over the ordered record. Reports how many events were scanned (honest coverage).

Added — follow-on study recommendations

  • A concluded study never dead-ends a thread its own findings opened. study("recommend") proposes follow-on studies a study's findings imply (proposals only — never auto-run), each seeded with the triggering findings; study("conclude") now also returns these. study("spawn", recommendation_id=…) approves one — creating the child study and a SPAWNED_FROM edge (child → source) carrying the reason + seed findings. The finding-signal → study-template mapping is an extensible trigger registry (recommend.py); the legal pack seeds disparate_treatment → judicial-bias, conflicting_dispositions → which ruling operative, ignored_favorable_evidence → due-process.

Added — blind spots + completion policy

  • study("confidence") — per-finding confidence + named blind spots: contested / low_depth_units worklists, coverage_by_lens (every registered lens with whether it ran + units examined — un-run ones are listed blind_spots, never silent), a recommended_next_escalation, and whether the study is settled.
  • study("set_policy") — a completion_policy (target_confidence, required_lenses, max_rounds) makes "done" a checkable contract: conclude now refuses (with explicit reasons) while the policy is unmet — a required lens un-run, or findings below target confidence — in addition to the synthesis gate. The same audited acknowledge_no_synthesis=true override records the skip + its reasons.

Added — leveled review: escalation rounds + lens registry

  • Spend review effort where it moves the needle. study("escalate", …) opens a ReviewRound and hands back only its targeted worklist — never a blind re-run: scope="contested"/"low_depth" returns the findings that need more reviewers; scope="uncovered" returns study chunks not yet examined by a given lens (a detectability sweep). study("next_review") claims a non-overlapping batch (punchcard keyed on the round); study("record_review") writes the EXAMINED coverage edge and, for a finding, casts the reviewer vote (updating confidence/escalation_state); study("close_round") counts the findings it produced. study("rounds") is the escalation history.
  • Lens registry (lenses.py, study("lenses")) — a lens is a named reviewer strategy (prompt + unit type). A generic open seam: core ships it empty; the legal pack registers contradiction / disparity / omission / quantitative / temporal. A registered-but-un-run lens becomes a named blind spot rather than a silent gap. Findings can carry an origin_round_id linking them to the round that surfaced them.

Added — semantic (cross-chunk) conflict detection

  • study("semantic_conflicts") — a deterministic scan that clusters non-neutral assessments by their classified element/topic and flags different chunks carrying opposing stances (one ruling supports, a conflicting ruling against) — the disparate-treatment / conflicting- disposition class same-chunk conflicts is structurally blind to. Honest coverage: reports checked (classified assessed chunks) vs skipped_unclassified, and when nothing is classified returns a loud note ("not looked", not a falsely-clean zero) recommending tag("classify") first.

Added — synthesis lifecycle + honest-completeness gate

  • A study can no longer be silently concluded while blind to cross-chunk patterns. The lifecycle is now score → verify → synthesize → conclude. study("synthesize", …) marks the ledger-wide pattern pass as run (the agent reads the whole ledger + records cross-chunk Findings); study("conclude") refuses (SynthesisRequiredError) until it has, unless acknowledge_no_synthesis=true records an audited SynthesisEvent (the skip is never silent). study("synthesis_prompt") returns what to hunt — a domain-neutral core list plus any registered domain addendum (the legal pack adds disparate-treatment / conflicting-disposition / ignored-argument cues). get_study surfaces synthesis_status + the synthesis-event trail.

Added — leveled review: Finding verification + confidence

  • Findings are gradeable by independent reviewers (study("verify", finding_id=…)). A cross-chunk Finding now accumulates independent verification votes (reusing the assessment VerificationEvent substrate; self-verification rejected). list_findings / get_study surface a reviewer-agreement rollup computed on read — reviewer_count, vote_tally, agreement, confidence (= max(0, net) × provenance factor), and an escalation_state (settled / contested / needs_more) that is also a routing label (MATCH (f:Finding:Contested)). Confidence is a function of recorded votes, not a single agent's assertion — the substrate the leveled control system is built on.

Added — cross-chunk synthesis (honest completeness)

  • Cross-chunk Finding unit (study("finding"/"findings")). A first-class reified node asserting a pattern over a set of chunks — the unit per-chunk assess structurally can't see (disparate treatment, conflicting rulings, …; a real run scored every chunk neutral while the strongest argument lived only in the JOIN of three). A Finding carries the same evidence axes as an assessment (stance / weight / provenance / rationale) but spans many chunks via SUPPORTED_BY edges (each must cite a real chunk); finding_type becomes a routing label (MATCH (f:Finding:DisparateTreatment)). Findings are a separate collection from the per-chunk assessment ledger — surfaced via study("findings") / get_study().findings, and cascade-deleted with the study. Generic: finding_type is free-text (the legal pattern vocabulary stays in the vertical). Foundation for the synthesis gate (next phase).

[0.0.12] — 2026-05-30

Release theme: multi-study routing — classify a corpus once into a domain element schema (a bundled legal vocabulary), then scope studies to the relevant element so many studies stop re-scanning the whole corpus. Scoping is advisory (rank-first) and carries a scope_coverage block — a speedup that is never lossy.

Added — multi-study routing (element classification)

  • Extensible element-discriminator registry (core primitive). A domain schema can now register element discriminators (register_element_discriminator(name, {value: label})) — the seam for classifying chunks into a controlled vocabulary so many studies route to their relevant chunks instead of re-scanning the corpus. Core stays domain-opaque: it routes opaque element tokens and validates them against the registered allow-list (valid_element_values() / element_label() — unknown tokens resolve to None so callers raise rather than silently match zero chunks). Adds the Classified/Unclassified/ Contested markers + a chunk.classify discriminator. (Inert until the classify pass + legal data pack land in later phases; the legal vocabulary ships as registered data, never hard-coded in core.)
  • classify.py — one-pass element classification + punchcard. next_unclassified hands out ready-but-unclassified chunks in reading order (claiming them per agent when agent_id is given), reusing the study punchcard via a new shared checkout.py keyed on a classify sentinel so classify and study fan-outs never collide. classify_chunk/classify_many write the validated element labels (add-only, recall-safe routing) + the canonical element_types_json record, and mark every chunk :Classified xor :Unclassified (exhaustive — the work-list never leaks). Two agents disagreeing add :Contested (never a silent clobber). study.next_unassessed's checkout logic was extracted to the shared checkout.py (behavior-preserving).
  • Bundled legal schema pack (kglite_docs.schemas.load_schema("legal")). An opt-in controlled vocabulary of ~18 court-case element types across three families — legal_role (holding, reasoning, judge_remark, settlement, disposition_order, …), legal_authority (statute, case_citation, …), and legal_evidence (testimony, documentary, …) — plus a detection rubric (rubric_text()) an agent reads to classify consistently. It's data only: the vocabulary registers through the generic seam, so the engine never names a legal term (boundary principle intact). Multi-label, jurisdiction-neutral (common + civil law; illustrative PT/PROJUDI mappings). A classified corpus then routes by label predicate: MATCH (c:Chunk:JudgeRemark).
  • Advisory element scoping on studies (element= on next_unassessed / study_ledger). Once a corpus is classified, a study can scope to an element type so it reads its relevant subset first instead of re-scanning the whole corpus — the multi-study win. It is advisory, never lossy: the full work-list/ledger is still returned, only reordered (in-scope element chunks first, then other-classified, then unclassified/not-yet last), so an analyst stops early as a visible choice and a misclassified chunk is deprioritized, never hidden (element labels are predictions, not ground truth like doc_id/section_id). An unknown/unregistered element= raises (InvalidEnumError) rather than silently matching zero chunks. (Hard-exclude is a deferred, audit-gated follow-up.)
  • Honest-coverage surfacing for element scoping. A scoped study_ledger (element=) now carries a mandatory scope_coverage block (in_scope/excluded_other_element/excluded_unclassified/excluded_total/ ready_total) with the enforced invariant in_scope + excluded_total == ready_total — so an early stop on a scoped study is an informed choice, and a non-zero excluded_unclassified is a standing signal that the scope has a blind spot. Also exposed as Corpus.element_coverage(element). document("map") / triage_map gains an elements breakdown + classified/unclassified/ contested counts; status() gains the same three. Corpus.element_consistency() audits the element labels against the canonical element_types_json (drift is observable, never silent).
  • MCP surface for classification (zero new nouns). Folded into the tag noun: tag("unclassified", …) (punchcard work-list), tag("classify", chunk_id, elements=[…], agent_id), tag("classify_many", items=[…]). The study noun's next/ledger gain an element= param (advisory rank + the scope_coverage block). The MCP server loads the bundled legal schema pack by default (build_app(schema_packs=…)), and _INSTRUCTIONS documents the "classify once → route many studies by element" happy path.

[0.0.11] — 2026-05-30

Release theme: agent assist (pre-enrichment) — a deterministic, zero-LLM triage layer so agents spend tokens on judgment, not mechanical work. Cheap content signals + structured-entity pre-tagging at ingest, plus a one-call corpus map for orientation. Every signal is additive, observable, and advisory — never lossy. (Local OCR with a modern engine is the deferred next release.)

Dependencies

  • Adopt kglite 0.10.10 (pin >=0.10.10). It ships the fixes for the two papercuts we offloaded (KG-1: a node property named label is readable again; KG-2: CONTAINS usable as a relationship type) — both of which we already dodged, so no code change was needed; full suite re-verified green (195) on 0.10.10. It also fixes a prefix+number id-coercion bug (a1/reader-1 could collide to the same node) that could affect user-supplied agent ids — a correctness reason to require it. The one 0.10.10 breaking change (Wikidata prefixed-id n.id string→int) does not touch us (our ids are non-coercible sha/uuid/composite strings).

Added — agent assist (pre-enrichment)

  • Deterministic content signals at ingest (FEAT — agent assist). Ingest now precomputes cheap, model-free triage signals on every chunk so agents spend tokens on judgment, not mechanical work — and it's never lossy (signals only label; no chunk is altered or dropped):
  • word_count / char_count (alongside the existing token_count).
  • content_kind — prose / table / list / code / sparse — as both a property and a label predicate (MATCH (c:Chunk:Table)), so an agent routes to (or past) tables, lists, and code without reading them.
  • quality_score (0..1) with a :LowQuality flag on prose/sparse chunks whose text looks garbled (bad OCR/encoding) — advisory.
  • :Boilerplate on verbatim-duplicate chunks (repeated disclaimers / pages), flagged but kept.
  • Structured-entity pre-tagging (regex, generic): dates, money, emails, URLs, and identifiers are extracted into an entities map (surfaced parsed on get_chunk()) and flagged as Has* label predicates (MATCH (c:Chunk:HasMoney)) so an agent jumps straight to the high-value chunks instead of reading everything. Recall-oriented + advisory; domain-specific entity types stay in a vertical, not core. All surfaced on get_chunk(); backward-compatible (re-ingest to populate).
  • document("map") / Corpus.triage_map() — one-call corpus orientation. Aggregates every signal above (chunk counts, the content_kind breakdown, boilerplate / low-quality, structured-entity coverage, embedding state, OCR-pending pages) plus a human summary, so an agent orients without reading the corpus and routes work via the matching label predicates (MATCH (c:Chunk:Table), :HasMoney, …). Scope with doc_id.

[0.0.10] — 2026-05-30

Release theme: scale & polish — single-writer guardrails (advisory lock + assess_many), opt-in summary-augmented chunking, result/detail ergonomics and an MCP surface review, plus a raw-text fallback so no readable page is silently dropped. End-to-end integration tests now exercise the 0.0.7–0.0.10 feature set together on the real embedder.

Fixed

  • Text pages no longer silently dropped when pymupdf4llm yields empty markdown. Some PDFs structure poorly enough that pymupdf4llm.to_markdown returns an empty page even though the page has extractable text; that page became a silent :Empty chunk (and, with no raster images, wasn't even flagged needs_ocr) — readable text lost with no signal. parse_pdf now falls back to raw PyMuPDF page.get_text() when the markdown is empty, recovering the text as a normal ready chunk. Honest coverage: content we can read is never dropped.

Added — scale & polish

  • Result/detail ergonomics + MCP surface review (FEAT-14). corpus.cypher() returns kglite's ResultView — now documented as iterable (for row in res, each row a dict), indexable (res[0]["col"]), len()-able, with .to_list() /.columns (a regression test locks this in). get_chunk() returns an AttrDict so both detail["section_id"] and detail.section_id work (still a plain dict otherwise). Surface review: the 13 nouns each stand; the one redundancy — summary("claim") — is soft-deprecated in favour of the first-class study flow (define → assess → ledger), which is richer, multi-agent, and verifiable (the action still works). Rationale recorded in docs/architecture.md.
  • Summary-augmented chunking (FEAT-11, opt-in). ingest(..., context_summary="…") (and document("ingest", context_summary=…)) prepends a document-level blurb to each chunk before embedding, so the vector carries global document context (mitigates cross-document speaker/source confusion à la contextual retrieval) while the stored chunk text stays clean (search hits are unchanged). The summary rides on the Document node and is applied by both inline embed=True and the later index() pass. No LLM in core — you supply the summary; default off is byte-identical to before.
  • Single-writer guardrails (FEAT-12). kglite-docs is single-writer; this makes that safe and ergonomic:
  • Advisory lock. Opening a .kgl while another live process already holds it now raises ConcurrencyError (a PID-stamped <db>.lock) instead of silently racing on save and corrupting. Same-process reopen (create → save → open) is allowed and a stale lock from a dead process is reclaimed; in-memory corpora take no lock.
  • study("assess_many", rows=[…]) / Corpus.assess_many() — assess a fan-out of chunks in one validated, batched write (and a single persist through the MCP layer) instead of N round-trips. All rows are validated before any write, so one bad row aborts the batch with nothing written.
  • Docs: a plain single-writer Concurrency section in docs/architecture.md (fan out reads, funnel writes through one process).

[0.0.9] — 2026-05-30

Release theme: document structure — a middle grain between document and chunk. Sections (from the PDF outline or headings), section-scoped studies, pinpoint char-span cites on assessments, and opt-in structure-aware chunking.

Added — document structure

  • Structure-aware chunking (FEAT-10). ingest(..., structure_aware=True) (and document("ingest", structure_aware=True)) starts a fresh chunk at every top-level heading and never packs — or overlaps — across one, so a chunk never straddles two sections and its heading attribution is exact. Size-based splitting within a section is unchanged (the token target is still honored). Opt-in and additive — the default packs greedily exactly as before; pairs with Section nodes (cleaner section boundaries) and pinpoint cites.
  • Pinpoint spans on assessments (FEAT-6). assess takes an optional quote and/or char_start/char_end — the exact passage an assessment rests on — validated against the chunk text (an out-of-range span or a quote not found in the chunk is rejected; quote alone is located to derive offsets). Surfaced per row in study_ledger (quote/char_start/char_end), so the evidence record carries pinpoint cites instead of just "somewhere in this chunk." Backward-compatible: rows without a span read back as ""/-1.
  • Section nodes + section-scoped studies (FEAT-9). Ingest now derives Section nodes — the grain between document and chunk — from the PDF outline (doc.get_toc()) when present, else from top-level heading boundaries (generic, all formats). (:Document)-[:HAS_SECTION]->(:Section)-[:HAS_CHUNK]->(:Chunk); each chunk carries section_id (+ a doc_type slot verticals can fill). document("sections", doc_id) / Corpus.list_sections() list them with per- section chunk_count; study("next"/"ledger", section_id=…) scope the work-list and evidence ledger to one section. IngestResult.section_count and get_chunk().section_id make the structure observable. Backward-compatible — re-ingest documents from before this release to populate sections.

[0.0.8] — 2026-05-30

Release theme: evidence integrity — make the assess/verify model legally defensible: record what was checked not just how strongly, let agents park blocked evidence, resolve cross-agent corrections to one current truth, and surface the disagreement.

Added — evidence integrity

  • study("conflicts", study_id) surfaces contested evidence (FEAT-8). Returns the chunks that have both a current supports and a current against assessment, each with its opposing rows split by side (most contested first) — the disagreement an orchestrator should review before concluding. Computed over the current set (latest-per-agent, non-superseded), so a correction that resolves a disagreement removes it from the list.
  • study("supersede", …) + current-by-default ledger (FEAT-5, BUG-4). Audit-preserving correction: supersede_assessment(old_id, …) records a replacement assessment linked to the one it corrects by an explicit (:Assessment)-[:SUPERSEDES]->(:Assessment) edge. study_ledger and its tallies are now current-by-default — a superseded assessment is hidden (each row carries a superseded flag), with include_superseded=True for the full history. This resolves the cross-agent ambiguity where a pre-filter's row and an analyst's correction both showed with no winner, without weakening multi-agent coexistence (only explicitly superseded rows drop out). The old assessment is never deleted — the correction trail stays legible.
  • Provenance axis on assessments (FEAT-4). Each assess now records provenance — what was actually checked (the basis), distinct from weight (the strength): primary_text (read the source — default), characterization (a paraphrase/summary, not the source), or scanned_unread (a scan no one actually read — provisional). It's a secondary label on the Assessment, surfaced per row in study_ledger, so a conclusion resting on characterizations or unread scans is no longer indistinguishable from one resting on primary text. verify takes an optional provenance recording what the verifier checked (stored on the verification event). Backward-compatible: assessments written before this release read back as primary_text.
  • deferred stance for evidence assessments (FEAT-7). Alongside supports/against/neutral, an agent can now assess a chunk as deferred — "read but can't judge yet" (an image-only / needs_ocr chunk, or a claim awaiting a source not yet ingested). Unlike neutral ("read, irrelevant"), deferred is counted distinctly in study_ledger tallies (deferred/deferred_weight) and keeps the chunk in the study("next") work-list so it resurfaces for a later pass — blocked evidence is parked, never silently dropped. Filter the ledger with stance="deferred" to list what's still blocked.

[0.0.7] — 2026-05-30

Release theme: honest coverage — every coverage-reducing decision (image-only pages, unembedded chunks, an unindexed corpus, a truncated ledger) is now observable in the return value instead of silently assumed.

Added — honest coverage

  • document("coverage") / Corpus.coverage_report() (FEAT-1): per-document
  • corpus extraction & embedding coverage — image_pages, low_text_pages, extractable_text_ratio, pending_ocr, unembedded/embedded, and a human-readable summary ("N image-only & L low-text pages need OCR; U chunks unembedded — search blind until index()"). Coverage is reported as data, never silently assumed.
  • document("status") / Corpus.status() (FEAT-2): one-call corpus snapshot (docs, pages, chunks, embedded/unembedded, image_pages, pending_ocr, studies) — the first thing to check. Pages now persist extractable_alnum at ingest; pages ingested before this release count as low-text until re-ingested.
  • compose_context() now reports searched_fraction (FEAT-3): the share of the corpus actually searchable (embedded chunks ÷ ready chunks). 1.0 = full coverage; < 1.0 means part of the corpus was invisible to the query.
  • study_ledger() reports total + returned and accepts doc_id scoping (BUG-3). The ledger caps at limit (default 200) but used to give no hint when it clipped the evidence — an orchestrator could draw a conclusion from a silently-truncated record. It now returns total (matches before the limit) and returned (rows handed back); total > returned means raise limit to see the rest. doc_id= scopes the whole ledger (rows + counts) to one document, mirroring study("next", doc_id=…).

Fixed

  • Retrieval over an unindexed corpus is now a loud signal, not a silent [] (BUG-2). search() / compose_context() used to return [] when nothing was embedded — indistinguishable from "this query genuinely has no matches," so an agent would wrongly conclude no evidence exists (amplified by 0.0.6's embed=False default). Now: a corpus with ready chunks but 0 embedded raises NotIndexedError (call index() / ingest(embed=True) first); a partially indexed corpus emits a UserWarning and exposes the gap via searched_fraction; a genuinely empty corpus still returns []. Behaviour change for library callers: searching a ready-but-unembedded corpus now raises instead of returning []. The MCP happy path (ingest → index → search) is unaffected.
  • cluster_chunks(algorithm="louvain") no longer pollutes the graph with phantom Chunk stubs. Bare CALL louvain() clusters the structural graph (Pages + the Document, not just chunks), so the resulting IN_CLUSTER edges vivified non-chunk endpoints as stub Chunk nodes. Louvain now runs only when a SIMILAR_TO chunk-similarity graph exists (its documented precondition) and filters results to chunk nodes; otherwise it falls back to embedding k-means — which is what the corpus does today, so the behaviour is the same minus the stray stubs.
  • Test runs are now warning-clean: the IN_CLUSTER stub warning is fixed at the source, and pymupdf's third-party SWIG DeprecationWarnings are filtered (tightly scoped in pyproject.toml so our own warnings stay loud).
  • Image-only PDF pages are now detected for OCR by text density, not "has any text at all" (BUG-1). pymupdf4llm emits a ==> picture … intentionally omitted <== placeholder for image regions, which previously made a scanned page count as ready and hid it from ocr_status(). A page with image content but fewer than OCR_TEXT_THRESHOLD (~120) real alphanumeric chars is now flagged needs_ocr. Pages also carry an image_block_count (for the upcoming coverage report). Honest-coverage: unreadable pages are observable.

Documentation

  • Confidentiality posture (FEAT-13): new Confidentiality page (+ README section) stating plainly that all parsing, embedding, and analysis run locally against a local .kgl — the only network call is a one-time bge-m3 weight download from HuggingFace; no document content is ever transmitted. Covers the benign "unauthenticated requests to HF Hub" message, automatic HF_HUB_OFFLINE once cached, and a fully air-gapped setup recipe.

[0.0.6] — 2026-05-29

Added — evidence-study workflow (new study MCP noun)

  • First-class evidence analysis. A validated multi-agent trial showed the prior workaround (stance smuggled into a tag name, weight overloaded onto tag confidence, rationale in a disconnected Summary, a 3-query stitch to rank) was the wrong shape. The new study noun owns the flow: define a question → assess each chunk (supports/against/neutral
  • 0–1 probative weight + rationale) → ledger (weight-ranked evidence
  • support/against tallies; stance= filters to one side) → verify (second agent: verified/disputed/duplicate) → conclude (a verifiable summary on the study) → list/get/reopen/delete. next is a resumable work-list of un-assessed chunks.
  • Reified Assessment nodes ((Chunk)-[:ASSESSED_AS]->(Assessment)-[:OF_STUDY]->(Study), (Agent)-[:AUTHORED]->): multiple agents can co-assess the same chunk; each assessment is independently verifiable. Append-only / latest-wins (no mutated-property hazard); stance/status/verification tracked as secondary labels for index-speed filtering.
  • No embeddings required. assess never touches the model (rationale is a plain property) — a whole study runs on an un-indexed corpus; agents iterate chunks via next. Recording is ~10ms/chunk.
  • Punchcard claiming on next. study("next", study_id, agent_id=…) atomically checks out the returned chunks (a Checkout node) and excludes chunks already claimed by others, so a fan-out of analysts gets disjoint batches and never overlaps. Claims auto-expire after 30 min and assessing releases implicitly; omitting agent_id is a read-only preview. (A blind two-agent trial had both analysts grab the same chunks; this closes that.)
  • Read-context + recorded spans for incoherent chunks. chunk("get", window=N) returns the N chunks before/after in reading order with text (via the NEXT_CHUNK spine), so an agent can interpret a chunk that doesn't stand on its own. When the neighbours were actually needed, study("assess", context_chunk_ids=[…]) records them as USED_CONTEXT edges: the ledger row surfaces the span (retrieval pulls the full relevant neighbours), and those context chunks are excluded from the work-list so no agent re-judges the same meaning.
  • MCP surface: 12 → 13 noun tools (revisit merges later). Server _INSTRUCTIONS gains an evidence-study happy-path; the study tool docstring is a copy-pasteable loop so agents need no prompt-side convention.

Fixed — tag ranking exposes confidence

  • tag("list") now returns each application's confidence (it was dropped, so the typed surface couldn't rank by it); tag("chunks") ranks by strongest confidence then tagger count. add_summary(embed=False) makes summary embedding opt-out (study conclusions default to no-embed).

Changed — embedding is now optional and explicit

  • Ingest no longer embeds by default. document("ingest") (and Corpus.ingest) now parse, chunk, and store without touching the embedding model — fast, no model load, no per-call-timeout risk. Real PDFs that took 48–90s to ingest now return in ~6s. Ready chunks are tracked as unembedded via the c.embedded property.
  • New document("index") action (Corpus.index) computes embeddings for ready-but-unembedded chunks, in length-sorted batches to cut bge-m3 padding waste. Run it after ingest to enable search; skip it entirely for non-semantic workflows (browse / cypher / tag / review / ocr / export / translate need no embeddings). One-shot still available via document("ingest", ..., embed=True).
  • index is bounded per call and loop-friendly. A blind-agent test showed that while ingest was now fast, indexing a whole multi-document corpus in a single call still ran ~100s (and a 144-chunk corpus ran long enough that the agent abandoned it) — the per-call-timeout risk had just moved from ingest to index. index now does at most ~30s of work (wall-clock budget, or max_chunks) per call, commits the partial progress, and returns pending > 0 with a hint to call again. The agent loops while pending: index(); no single call exceeds the budget (+ at most one batch). max_seconds=None drains everything in one pass (used by CLI --ingest preload, where there's no timeout).
  • Ingest results now carry embedded count + a hint pointing at index when chunks are unembedded.

Fixed

  • Tool-driven ingest now persists to disk. The long-lived MCP server previously only saved when launched with --ingest; interactive document("ingest") mutations were lost on restart. Now saved at the tool boundary (ingest/index) plus a save-on-shutdown backstop for all mutations.
  • Embedding lifecycle is tracked by the c.embedded property (with an additive :Embedded label for browsing) rather than a removable :Unembedded label. This was originally a workaround for a kglite multi-label read bug — MATCH (n:Label) over-reported after remove_label — which we reported and kglite fixed in 0.10.6 (root cause was read-side fast-paths consulting only the primary-type index, not a stale index as we first guessed; no data corruption, no .kgl migration). The property-based design is version-independent so we kept it; removable labels / swap_label now work correctly regardless.

Dependencies

  • Bumped kglite>=0.10.9 (multi-label read fix from 0.10.6; FxHash query perf from 0.10.7: multi_where −38%, group_by −23%, where_scan −21%; four Cypher-dialect fixes in 0.10.8 we reported buildingstudy—labels()/properties on acollect()[0]node, parameters insideEXISTS {}, node-expression inline-map values, andDETACH DELETEskipping NULL vars; and the 0.10.9 self-healing id-index that makesMATCH (n {id: X})/ MERGE lookups O(1) — a free win for the study workflow's many id lookups) andmcp-methods>=0.3.40(fixesgraph_overviewforwarding the droppeddescribe(limit=)kwarg, andcypher_queryfailing-> strvalidation on kglite'sResultView). Both escape-hatch tools now work on kglite 0.10.x. The latent over-reporting in the otherswap_label` lifecycles (summary verification, OCR, review kanban, translations) is fixed by the 0.10.6 upgrade.

Performance

  • Background bge-m3 warm-load at server boot (off-thread), HF_HUB_OFFLINE when weights are cached, and cooldown=0 for the server — the model load (~8s) and HF-hub network checks no longer land on the first ingest/search call. --no-warmup skips it for non-embedding deployments.

[0.0.5] — 2026-05-28

Changed (MCP surface — agent-facing)

  • Collapsed 45 standalone MCP tools into 12 CLI-style noun dispatchers. Each dispatcher takes an action verb as the first positional arg:
document(action, ...)   ingest | list | get | export | compare
chunk(action, ...)      get | similar
search(query, mode=…)   hits (default) | compose
summary(action, ...)    add | verify | list | ground | claim | consensus
tag(action, ...)        add | remove | list | chunks
agent(action, ...)      upsert | get | list | activity
review(action, ...)     enqueue | enqueue_chunks | claim_next | claim |
                        unclaim | complete | list | get | stats
ocr(action, ...)        status | pending | submit
cluster(action, ...)    run | get | list | export
translate(action, ...)  add | list | assemble

Plus cypher_query and graph_overview from mcp-methods. An agent reads the methodology skill and copies verb-noun pairs straight into tool calls (document("ingest", path=...), summary("verify", id=..., verdict=...)) — patterned like git branch list, not 45 standalone functions to scan.

The Python Corpus API is unchanged — the dispatch only lives in the MCP shim.

Added (agent onboarding)

  • Corpus.compare_documents(doc_a, doc_b, queries=[...]) — side-by-side cross-document retrieval. For each query, returns the top hits from each document plus a budgeted merged context bundle. Exposed as document("compare", ...) over MCP.
  • First-contact instructions= on the FastMCP server. Connecting agents receive a ~250-word orientation on initialize covering the CLI surface and the canonical happy path.
  • 4 methodology skill files registered as MCP prompts via mcp-methods' SkillRegistry:
  • /00-start-here — overview + the canonical happy path
  • /analyze-documents — canonical pipeline for "given N docs + a task"
  • /compare-documents — the two-doc comparison idiom
  • /cross-checked-review — write/verify/ground flow for trustworthy summaries

Skills are loaded by the agent on demand (/<skill-name>) — not preloaded into context.

Removed (redundant — covered elsewhere)

  • register_agent → agent("upsert", ...)
  • link_verification → niche; use cypher_query
  • record_view → implicit when passing agent_id to search / chunk("get", ...)
  • untag_chunk → tag("remove", ...)
  • ingest_dir, ingest_text → document("ingest", directory=...) / document("ingest", text=..., title=...)

0.0.4 — 2026-05-28

Changed (internal data model — public API unchanged)

  • Adopted kglite 0.10.5 multi-label nodes. Categorical / lifecycle state — Agent.role, Agent.kind, Chunk.status, Summary.verification_status, Tag.kind, Translation.status, and ReviewTicket lifecycle — now lives as secondary labels on the relevant node ((a:Agent:Reviewer:LLM), (c:Chunk:Ready), (s:Summary:Verified), etc.). Property scans are replaced with MATCH (n:Label) predicates, which kglite indexes natively. Cross- type predicates like MATCH (n:Reviewed) return any reviewed thing in the graph regardless of primary type — useful for governance queries.

The Corpus Python / CLI / MCP surface is unchanged: callers still pass status="verified" / role="reviewer" / verdict="reviewed" strings. A new schema.label_for(discriminator, value) helper maps user-facing strings → canonical PascalCase label names at the boundary. Free-text discriminators (notably Agent.role) get slug → PascalCase conversion: "fact-checker" → :FactChecker.

Labels are maintained atomically via a new Store.swap_label primitive: every state transition (verify_summary, submit_ocr, claim_review, complete_review, mark_translation_reviewed) removes the old label and adds the new one in one call.

Event-sourced models (VerificationEvent, ReviewEvent) still carry the immutable audit trail. The labels denormalise "current state" for O(label-index) reads instead of "scan events, pick latest".

  • 7 new tests covering label round-trip, swap mutual-exclusion, save/reload survival, and cross-type label predicates.

  • kglite>=0.10.5 required (was >=0.10.4).

Breaking — raw Cypher only

If you wrote raw Cypher against corpus.cypher(...) and filtered on Summary.verification_status, Chunk.status, Agent.role, Agent.kind, Tag.kind, or Translation.status, switch from WHERE n.prop = '...' to MATCH (n:NodeType:Label). Example:

-- before
MATCH (s:Summary) WHERE s.verification_status = 'verified' RETURN s
-- after
MATCH (s:Summary:Verified) RETURN s

Label names use PascalCase: Verified, Disputed, NeedsRevision, Stale, Unverified, Ready, NeedsOcr, Empty, LLM, Human, Service, Reviewer, Writer, FactChecker, etc. The Cypher escape hatch tool in the MCP server still works the same way — the agent just writes label predicates now.

.kgl files created by v0.0.x still load on v0.0.4 — labels won't exist on those nodes (so label-filter queries against old corpora return empty); re-ingest to populate labels.

0.0.3 — 2026-05-28

Added

  • Agent nodes carry a reusable template. The Agent node was previously identity + counters; it now also holds role, system_prompt, model, tools (list), context (free-form JSON dict), and description. The graph IS the registry — orchestrators fetch an agent's loading context with get_agent(agent_id), use the fields to launch the actual LLM call, and every subsequent graph write under the same agent_id attributes back to the template.

New methods on Corpus:

  • upsert_agent(agent_id, *, kind, model, role, system_prompt, tools, context, description) — field-level merge.
  • get_agent(agent_id) — returns full template + counters; tools and context come back hydrated (real list / dict, not JSON strings).
  • list_agents(role=..., kind=...) — discovery, with filters.
  • agent_activity(agent_id, *, target_id=None) — bucketed rollup of everything an agent has done (views, summaries, tags, translations, review/verification events). Pass target_id to scope to one chunk.

Old register_agent (the lazy on-first-use path) is preserved and now explicitly does not clobber template fields when the agent already exists. add_summary / tag_chunk / etc. still lazy-register if you haven't upserted first.

New types: AgentConfig, AgentActivity.

New MCP tools: upsert_agent, get_agent, agent_activity (existing register_agent / list_agents keep working; list_agents gained role= / kind= filters).

9 new tests covering create → update merge → preserves-on-lazy → scoped activity rollups → realistic orchestration round-trip.

Example::

  corpus.upsert_agent(
      "reviewer-strict",
      role="reviewer", model="claude-sonnet-4-6",
      system_prompt="You are a strict fact-checker...",
      tools=["check_grounding", "verify_claim"],
      context={"strictness": "high", "min_citations": 2},
  )
  # Later — orchestrator side
  cfg = corpus.get_agent("reviewer-strict")
  anthropic.messages.create(
      model=cfg["model"], system=cfg["system_prompt"], ...
  )

0.0.2 — 2026-05-28

Changed

  • mcp + mcp-methods moved into core dependencies. They were previously gated behind the [mcp] extra, but mcp was already pulled in transitively by kglite anyway, and mcp-methods is only ~17 MB — rounding error against pymupdf (30 MB) and onnxruntime (80 MB) which are core. The MCP server is the headline feature; gating it behind an extra was friction with no real savings.

Install is now just pip install kglite-docs. Old install commands with [mcp] keep working (pip warns, doesn't fail).

CI / tooling

  • Bumped all actions/* to versions supporting Node.js 24 ahead of GitHub's 2026-06-02 deprecation of Node 20.

0.0.1 — 2026-05-28

First public alpha. The core agent-first PDF knowledge-base API is in place: ingest → chunk → embed → search → enrich → cluster → review, served over Python, CLI, and MCP. Everything is exercised by 81 unit + integration tests; a real end-to-end Sonnet workflow demo is included.

Added

  • Multi-format ingest. PDF, DOCX, PPTX, MD, HTML, TXT, and common image formats (PNG/JPG/TIFF/WebP/BMP), all flowing into the same Document → Page → Chunk graph.
  • Token-aware paragraph chunking at ~512 bge-m3 tokens with 64-token overlap; never crosses a page boundary.
  • BAAI/bge-m3 embeddings via ONNX (CLS-pooled, 1024-dim, 8192-token cap), inherits kglite's cool-down lifecycle for warm-call latency.
  • Semantic search + compose_context(query, max_tokens=…) — budgeted, ranked, prompt-ready bundles for agents.
  • Enrichment with cross-checking. add_summary / verify_summary enforce that the verifier is a different agent. Status is event-sourced via VerificationEvent so we keep a full audit trail.
  • Tagging via reified Tagging nodes so multiple agents can tag the same chunk distinctly with (by_agent, created_at, confidence).
  • Agent identity propagation. Lazy-registered Agent nodes; search / get_chunk accept agent_id= to bump Chunk.view_count and record View nodes when the query context is worth keeping.
  • Scanned-page OCR loop. list_pending_ocr → agent reads PNG → submit_ocr(page_id, markdown) re-chunks + re-embeds.
  • ocr_status() — coverage summary across the corpus, per-doc detail with a pending-fraction. Exposed as Python / CLI / MCP.
  • kglite-docs ocr-do — CLI subcommand that drives the OCR loop with any agent command containing {image} (e.g. claude -p --image …).
  • Clustering. cluster_chunks(algorithm='louvain'|'kmeans'|'dbscan') with Cluster + IN_CLUSTER graph state; most_connected_cluster for synthesis use cases.
  • Quality / anti-hallucination. check_grounding(summary_id) scores per-sentence support; verify_claim(claim_text, …) finds best-supporting chunks.
  • Translation layer. Translation nodes per chunk × language; assemble_translated_document stitches back with fallback to source.
  • Export. export_document / export_cluster / export_summary / export_bundle to Markdown, DOCX (python-docx), or PDF (ReportLab).
  • Review queue (kanban). Event-sourced ReviewTicket → ReviewEvent audit trail. claim_next_review / complete_review with verdict + accuracy + authenticity + tags. Process-local lock protects against double-claim within a process.
  • MCP server. kglite-docs-mcp --db kb.kgl exposes 30+ typed tools plus cypher_query / graph_overview escape hatches via mcp-methods.
  • CLI. kglite-docs ingest|search|list|cluster|show|ocr-status|ocr-do.
  • Context manager on Corpus. with Corpus.open(path) as c: auto-saves on clean exit, skips save on exception so a partial mutation isn't persisted.
  • Typed errors. KgliteDocsError base + concrete subclasses (IngestError, UnsupportedFormatError, MissingSourceError, ReviewConflict, SelfVerificationError, InvalidEnumError, GroundingError, ConcurrencyError).
  • Typed args + returns. Literal[…] for verdicts, statuses, depths, target kinds, algorithms, tag kinds; TypedDict return shapes (SearchHit, OcrStatus, ReviewTicketDetail, …) for IDE autocomplete and mypy --strict.
  • End-to-end Sonnet workflow demo (demos/workflow.py): ingest → cluster → Sonnet summarises → Sonnet drafts an article with [chunk_id] back-references → second Sonnet pass fact-checks → verifications persist as VerificationEvent nodes.
  • Docs. Getting-started, architecture, workflows, performance, publishing, troubleshooting, contributing — all in docs/.
  • CI + release. GitHub Actions workflows for ruff + mypy + pytest on Py 3.10–3.13 × macOS/Linux, plus a trusted-publisher PyPI release pipeline triggered on v* tags.

Dependencies

  • Requires kglite>=0.10.4. (Earlier 0.10.3 hit two upstream bugs we filed; both fixed in 0.10.4 — see "Bug workarounds peeled" below.)
  • Requires mcp-methods>=0.3 (was previously in the [mcp] extra; moved to core deps in 0.0.2).
  • bge-m3 ONNX weights download to ~/.cache/fastembed/ on first use (~2 GB, one-time). Set HF_HUB_CACHE to reuse an existing HF cache.

Bug workarounds peeled

Two kglite bugs we hit and reported during this build, both fixed upstream in 0.10.4:

  • add_nodes(Chunk, …) invalidating the id index used by set_embeddings. Pre-0.10.4 this silently wiped all prior chunk embeddings on the second-and-later document ingested into a corpus. We carried a save+reload workaround in pipeline.py (since removed).
  • mmap_vec panic on the second String SET on the same node. Forced us into event-sourced verifications and reviews. The kglite fix lets us mutate Strings normally now, but we kept the event sourcing — the audit trail is a better model on its own merits.

Filed in the kglite inbox at KGLite/inbox/read/2026-05-28-from-kglite-docs-multi-file-ingest-loses-old-chunk-embeddings.md.

Known limitations

  • Single-writer per .kgl file. kglite is process-local; concurrent writers to the same file aren't supported. Corpus holds a Python- level lock for the review queue's claim path, which covers multi-threaded but not multi-process scenarios.
  • No OpenAI/Cohere embedder adapter shipped. The EmbeddingModel protocol is documented (dimension, embed, optional load/ unload), but only BgeM3Embedder is wired by default. Writing an alternative is straightforward.
  • PDF export is "good enough" ReportLab Platypus, not pixel-perfect. Use the MD or DOCX path for richer output, or pipe MD through pandoc.
  • find_consensus and link_verification are experimental. They work but the shape of the API isn't as well-considered as the rest. Subject to change in 0.x.

Test posture

  • 81 unit + integration tests, ~1s run time with the stub embedder.
  • @pytest.mark.embed tests use the real bge-m3 model (skipped in CI to keep things fast; run locally with the model cached).
  • End-to-end Sonnet workflow demo proves the agent path beyond unit tests.