2026-09-01-conversation-ingest

2026-09-01 — conversation ingest pipeline §

The author does his long exploratory thinking on claude.ai, not in the vault. He asked for a way to get a finished conversation out of the web app and into the garden, without the web app trying to write Obsidian notes it cannot see.

Two skills and one shared contract:

Piece	Path	Side
garden-dump	claude-web-skills/garden-dump/SKILL.md	claude.ai
The format	claude-web-skills/garden-dump/references/format.md	both
garden-ingest	.claude/skills/garden-ingest/SKILL.md	Claude Code
The linter	.claude/skills/garden-ingest/scripts/lint_dump.py	Claude Code

A dump lands in private/dumps/, which is gitignored and outside the vault.

The design idea §

The web app cannot see the vault, so anything it writes about the vault is a guess. The split follows from that: claude.ai transcribes, Claude Code decides. The dump carries the author’s words verbatim and marks every claim by origin. Every vault-facing act — the filename, the duplicate check, the wikilinks, the frontmatter — happens on this side, against the real vault.

The load-bearing rule is asymmetric compression. His words are transcribed; the model’s own exposition compresses to the bare claim. The invariant is countable, so the linter can enforce it: at least as many author quotes as idea units.

What the critique changed §

Four reviewers attacked a first draft of the format. Five findings changed the design rather than polishing it.

1. The dump would have published. It is markdown with no publish key, and the obvious place to put it is next to the notes. It now lives outside the vault, carries publish: false as a second defence, and the linter refuses a dump found inside digital-garden/.
2. A ## Sensitive / personal section is the one artifact this pipeline must not produce. Claude/index.md forbids an index of what is sensitive, and “flag it, describe it minimally” is an instruction to write exactly that. Personal passages now leave the file entirely, and ## Coverage keeps a bare count. The linter rejects anything but a count on that line.
3. ### 1. is not an identity. Numbering restarts in part 2, shifts when an idea is dropped, and cannot express a cross-part reference. Ideas now carry a slug id, and related and supersedes reference ids.
4. “Lossless on the author, lossy on Claude” would not have survived. It is a metaphor, and it asks a model to invert its training. It became two named operations plus the countable quote invariant.
5. The summary sections primed the model into summary mode. Writing the arc first sets a register that bleeds into every idea below. ## What this was about and ## Turning points are now emitted last, and the turn ledger is emitted first.

The turn ledger is the other structural answer to the same problem: one line per author message, each pointing at an idea id. A post-hoc coverage tally is unfalsifiable; a ledger the linter can join against is not.

What was rejected §

• A summarize-or-dump mode switch. The author asked for one, and it turns out to be the wrong axis. Lossless on his words and lossy on the model’s is right for every conversation, so nothing needs choosing.
• A status field. It collides with growth, the vault’s one classification axis, and would have been copied into frontmatter. Renamed to confidence, with an explicit rule that it never maps onto growth.
• Claim-shaped titles as the default. They are a minority form here, and forcing one onto a concept note produces a title nothing will ever link to. The dump now carries form, and garden-ingest picks the title shape from it.
• Relying on the prompt for ASCII punctuation. Negative instructions decay across a long generation. The rule stays, but the linter is the guarantee.

What the second pass corrected §

The first draft of all four files was written, then verified by three reviewers who read them as a consumer, as a linter and as a runnable procedure. They found defects the design critique could not, because these only appear once the files exist.

1. The worked examples used a real note. Every example in garden-ingest and in the sample dump was built around Welds trade serviceability for production cost, which is a published note in Notes/. An agent copying the example verbatim would have created a duplicate of it in Inbox/. All examples moved to a sourdough theme, checked against the whole vault for collisions.
2. Two working directories, one stated. The skill said “run from the vault root”, but half its commands are repo-root relative. From the stated directory, step 1 could not find the dump at all. Both roots are now named, with a table of which commands belong to which.
3. ls private/dumps/*.md aborts in zsh when the directory is empty, before 2>/dev/null can suppress anything. Replaced with find.
4. The linter rejected dumps that follow the contract. It treated “the author’s words” as “a line starting with >”, but the format requires verbatim text in five other places: the turn ledger, search terms, constraints, to-dos and retrieved input values. One curly apostrophe from a phone keyboard blocked the whole ingest, and the skill forbids hand-repair, so the dump was unrecoverable. The exemption is now section-aware and fence-aware, and covers any straight-double-quoted span.
5. The gate did not catch the worst dump. A dump that quoted a personal message in the ledger and wrote a topic list into omitted as personal passed clean, while omitted as personal: none failed. Both are fixed, and the linter now also verifies the ## Coverage counts against the file rather than trusting the model’s self-report.
6. The memory loop was never closed. Step 8 wrote ingested-dumps.md and no step ever read it, so a second dump of the same conversation was exactly as undetectable as before. Step 1 now reads it first.
7. An append could publish the conversation. 245 of the 248 notes in Notes/ carry publish: true. The skill proposed appends into them with no publish check. It now reads the target’s key first and says so in the table, and refuses Journal/, Projects/, Research/, misc/ and Organization/ outright.

Smaller ones, all real: grep -ix treats a note title as a regular expression and 15 titles contain metacharacters, so every filename check is now grep -F; three of the five style-tell patterns could never match; zip -x '.*' did not exclude .DS_Store; and guard rail 2 forbade the publish: true that step 8 requires.

What is unverified §

• No dump has been produced yet. The format, the skill and the linter were built and tested against a hand-written example (claude-web-skills/garden-dump/references/example.md), not against real output from claude.ai. Expect the first real dump to break something.
• .md is not a documented downloadable output type on claude.ai. The listed types are .xlsx, .pptx, .docx and .pdf. The sandbox is a general filesystem so writing markdown works in practice, but it is not a documented contract. The skill falls back to printing the file in a code block.
• Skills on the free plan. Anthropic’s help centre lists Free; the platform docs do not. Pro and above is the safe claim.
• Claude/work/ingested-dumps.md does not exist yet. garden-ingest creates it on the first real ingest. It records conversation ids and note filenames only, never content, because Claude/ publishes.

The publish audit was run at the end of this session. The two high-severity findings (Research/Research.canvas, publish.css) are pre-existing and unrelated: nothing in this session wrote to the vault outside this log.