Skip to content

improvement: plugin interface v2 - #80

Merged
sidkmenon merged 26 commits into
mainfrom
plugin-host-cursor
Oct 3, 2026
Merged

sidkmenon merged 26 commits into
mainfrom
plugin-host-cursor

Conversation

@sidkmenon

@sidkmenon sidkmenon commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

significantly simplify the plugin interface. split into two types, now just defined by WASM:

  1. The classifier plugin implements the generated GuestClassifier trait; it determines if and how an entire file is (1) collapsed and (2) skips structural diffing:
use diffr_plugin_sdk::prelude::*;

struct MyClassifier;
impl ClassifierGuest for MyClassifier {
    type Classifier = Self;
}
impl GuestClassifier for MyClassifier {
    fn new(options: String) -> Result<Self, String> { Ok(Self) }
    fn classify(&self, file: FileEntry) -> Result<Classification, String> {
        Ok(Classification { tags: vec![Tag::Custom("schema".into())], hidden: None })
    }
}
export_classifier!(MyClassifier);
  1. A shape plugin implements the generated GuestPlugin trait; it will be reminiscent of golang's astutil.Apply, for those familiar. Shape plugins determine the presentation of individual AST folds in an open file (e.g. collapsing + replacing w pseudocode, or them grouping together, and so on):
use diffr_plugin_sdk::prelude::*;

struct MyPlugin;
impl Guest for MyPlugin {
    type Plugin = Self;
}
impl GuestPlugin for MyPlugin {
    fn new(options: String) -> Result<Self, String> { Ok(Self) }
    // cursor is a host-provided interface over the matched ast tree(s); phase indicates preorder/postorder.
    async fn visit(&self, cursor: &Cursor, phase: Visit) -> Result<bool, String> {
        // Inspect this node and edit it through the host cursor.
        Ok(true)
    }
}
export_shape!(MyPlugin);

it all ties together like this:

sequenceDiagram
    participant C as UI / API consumer
    participant D as diffr
    participant G as git
    participant K as Classifier plugin
    participant E as Diff engine
    participant S as Shape plugins

    D->>G: List the comparison's changed files
    G-->>D: Paths, blobs, statuses
    loop Each changed file
        D->>K: classify(file)
        K-->>D: Tags, and whether to hide the file
    end
    D-->>C: start record: the tagged manifest
    par Each file, on --jobs workers
        D->>G: Read both sides
        D->>E: Diff
        alt Hidden or generated
            E->>E: Line diff
        else
            E->>E: Parse with tree-sitter, diff ASTs (difftastic)
        end
        E-->>D: Aligned regions and folds
        loop Each shape plugin, in plugins.order
            D->>S: visit(cursor, pre | post), once per region
            S->>D: cursor edits: cut, join, link, collapse, label
        end
        D-->>C: file record, as soon as it is done
    end
    D-->>C: complete record
Loading

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
src/git/diff.rs enumerates a comparison with gix: revisions, the index,
the working tree and the empty tree, with rename detection and pathspecs.
tests/git_comparison.rs checks the manifest, name-status and stat output
against git itself.

The session in src/git.rs is written against the plugin host that the
next commit lands; this commit does not build on its own.

AI assistance: developed with Claude Code and OpenAI coding agents.

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
…ed cursor

Contract: one WIT package in crates/diffr-plugin-sdk/wit defines two
worlds. A shape plugin is a constructor and visit(cursor, phase); the
host walks each file and calls it per node, pre and post. A classifier
returns tags and an optional hide reason per changed file. Both read the
repository through a typed git interface (check-attr, cat-file) backed
by gix. The native plugin path, the SDK's own tree, drafts, deferred
enrichment and the annotations stream are gone.

Host: src/plugin/cursor.rs owns each file's region trees, visibility,
ids and walk position and edits protocol::Region directly; the cursor
resource's methods are table lookups into it. src/plugin/wasm.rs runs
--jobs workers, each a thread with one Store holding an instance of
every enabled plugin; a file is classified and shaped by one worker,
every plugin in order, while files interleave when a plugin awaits I/O.
Pipeline::from_config is the only constructor. Guest output is prefixed
[plugins].

Configuration: a shape plugin's fold queries are files declared per
language in its plugin.toml; Config::compile reads them without running
a plugin, so query errors come before WebAssembly starts. [classifier]
names the one classifier, the bundled one by default. Its hide and
hide_deleted options replace the hide-files plugin, and the tag rules
move from src/tags into it. A hidden file is diffed by line, runs no
shape plugin and is written collapsed behind its reason.

Bundled plugins: plugins/shape/<name>/rust and plugins/classify/rust
form a separate wasm32-wasip2 workspace (plugins/workspace), and each
folder ships a prebuilt plugin.wasm that build.rs embeds. cargo xtask
build-plugins rebuilds them; test-plugins runs the component suites.

Runtime: wasmtime 48 with component-model-async; tokio and
async-channel for the worker queue.

Tests: tests/tags.rs runs the classifier end to end; tests/wasm.rs
compares bundled and external components; tests/wasm_concurrency.rs
checks that files overlap across awaits and across workers.

AI assistance: developed with Claude Code and OpenAI coding agents.

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Late annotation records and --stream-annotations are gone from the
binary, so the contract is version 3 only.

AI assistance: developed with Claude Code and OpenAI coding agents.

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Files stream as they finish, so a tag-priority sort before diffing does
not fit; a consumer that wants an order has the manifest.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
src/run.rs replaces protocol/stream.rs. git::list and git::standalone
each return a Listing of the two snapshots and the changed files, with a
blob source carrying the shared repository handle it reads from.
run::classify tags a listing; run::stream builds the diff pool, diffs,
projects and shapes each file on the workers, and writes records in
completion order. The only struct left is the private Shared bag the
tokio tasks need. DiffSession, Differ, FileJob, pool(), write and
write_file are gone.

The CLI parses flags into a merged Config (-U and the limit flags) and
run::Options (syntax, ignore-comments), builds Params and Pipeline by
their own constructors, and maps Ended to an exit code. Classification
is a step the repository path takes and --no-index skips, since the
bundled classifier would hide sample_files/ as vendored.

--no-index now reads its two files through the same loader as repository
files: NUL means binary, otherwise strict UTF-8. A UTF-16 file is a
not_utf8 record rather than transcoded. The old loader is unused after
this commit and is deleted in the next one.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
main::diff_file, the UTF-16 and Latin-1 heuristics in files.rs, the
.gitattributes check, FileArgument and two unused engine parameters go,
with the tree_magic_mini and encoding_rs dependencies.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
There is no id 0. get returns a region-view, a top-level region's parent
is none, and the walk visits regions only, so visit never sees a file
node and the bundled plugins lose their root guards. cut-file and
includes-file are gone from move-error: id 0 is an unknown id like any
other. set-collapsed and set-label act on regions only; a file's own
visibility is the classifier's hide reason alone, and a binary diff runs
no plugin. Bundled components rebuilt.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
src/plugin/classify.rs owns the classifier: one component, one Store, one
current-thread runtime, asked about every listed file on the calling
thread. Pipeline is shape plugins only; a worker instantiates no
classifier and Job is one kind. The CLI classifies the listing before it
builds the pipeline, so tagging no longer waits for every shape plugin to
be instantiated on every worker. The engine, store state and linker are
shared between the two.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Cursor::new positions the cursor on the file's first region and hands
the sides back for a file that has none, such as an empty one; the
pipeline then runs no plugin on it. The walk sets the position before
each visit, so id is a plain u32 that always names a region.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Workers no longer pull from a shared queue. Each has a mailbox holding
one file and a count of files in flight; Pipeline::run gives a file to
the least-loaded worker whose mailbox has room, choosing and counting
under one lock. A worker whose loop is alive takes the file at once,
including one whose files are all parked on I/O; a worker blocked in
guest compute cannot, so its mailbox stays full and dispatch moves on.
Only when every mailbox is full does a file wait on the least-loaded
worker.

On the worker, every file is a task spawned on the Store's own event
loop (Accessor::spawn), so wasmtime schedules the guest calls and the
select loop with its poll and yield is gone. A trap fails that file's
run rather than the worker. async-channel and futures are no longer
needed.

AI assistance: developed with Claude Code (Claude Fable).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
The summarizer calls out from inside WebAssembly through wasi-http, so
the host's own client was unused. It also enabled rustls's aws-lc-rs
backend next to wasmtime-wasi-http's ring, and rustls refuses to pick
between two, which panicked the worker on any HTTPS request.

AI assistance: developed with Claude Code (Claude Fable).

Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
rust-toolchain.toml, rust-version and the pinned CI toolchains move to
1.99.0. Its dead-code pass no longer sees through lazy_static's
initializer, so the version string is a std LazyLock.

AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
wasmtime, wasmtime-wasi and wasmtime-wasi-http go to 49.0.2 with the
same features; no host code changes. Wasmtime 49 needs Rust 1.96.

wasmtime-wasi-http 49.0.2 aborts an outgoing request's connection as
soon as the guest drops the request's transmission future, and the
wasip3 http_compat conversion drops it at once, so every HTTPS response
body failed with HttpProtocolError. The summarizer now builds its WASI
request itself and awaits that future after reading the body. Bundled
summarize component rebuilt.

Bundled components rebuilt with Rust 1.99 and this Wasmtime.

AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
The parser, diff, projection, configuration and plugin cursor now build
as a library with no async runtime, threads, git, component runtime or
terminal. The cursor speaks plain core types: a region view, a move
error, row summaries and the ids an edit made. The diffr package keeps
the CLI, git, the stream and the wasmtime host; its bindings file is the
one place the contract's records meet the engine's types, with a From
impl per record.

AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
present() now takes the plugin run as an async closure over the sides,
so the engine can recount visibility without knowing the wasmtime
pipeline. The engine's modules the diffr package no longer reaches are
private again.

AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
The classifier's regex dependency kept only the Unicode tables it needs
and so lost every perf feature: no literal prefilter, no lazy DFA, the
backtracker for every path against Linguist's joined vendor pattern.
That cost about a millisecond per file, 5.6 s before the first record
on a 5,600-file comparison. With perf on it is 15 µs per file and
0.1 s. The component grows from 0.9 MB to 1.4 MB.

AI assistance: developed with Claude Code (Claude Fable).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
check-attr built a fresh attribute stack per call, loading the index
and every .gitattributes file again for each path. The Store keeps one
stack and queries it per path.

AI assistance: developed with Claude Code (Claude Fable).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
AI assistance: developed with Claude Code (Claude Fable).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
The shape plugin's export macro is export_shape!, next to the
classifier's export_classifier!; the prelude carries both and the
classifier's types, so either kind of plugin starts from one import.
docs/plugin.md gets the sequence diagram back for the current flow and
the links it was missing.

AI assistance: developed with Claude Code (Claude Fable).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
@sidkmenon
sidkmenon marked this pull request as ready for review October 3, 2026 10:47
@sidkmenon sidkmenon changed the title wip: plugin interface v2 improvement: plugin interface v2 Oct 3, 2026
@sidkmenon

Copy link
Copy Markdown
Contributor Author

upside: there is a HUGE performance improvemnet in here as well; for large diffs:

image

i.e. abt 30x improvement in time-to-first-diff and 5x improvement in total time taken :)

Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
The root package's all-languages forwarded straight to diffr-core's,
leaving its own lang-* features off, so a test checking
cfg!(feature = "lang-apex") disagreed with the binary.

Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA
Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
@sidkmenon
sidkmenon enabled auto-merge (squash) October 3, 2026 11:00
Case A timed out at thirty seconds on the x86_64 Linux and coverage
jobs while the passing coverage run took eighty seconds for all five
cases. The timeout only guards against a hang, since the server's
notifications enforce the ordering under test.

AI assistance: developed with Claude Code (Claude Opus).
Claude-Session: https://claude.ai/code/session_01FKkBihPSkFktKruXdCnsrA

Agent-Session: 40d62209-6ba3-4318-8ef9-b1287f57a712
Agent-Session: 01a0ff62-10be-7770-8692-b9540adcd4a3
Agent-Session: beda0994-f79e-4d8c-86d2-1e91557d62cc
Agent-Session: 01a10126-800c-7922-a418-bb477c280cbd
@sidkmenon
sidkmenon merged commit c82fe61 into main Oct 3, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant