Testing Plugins Against a Host Contract

This page answers one task: external developers write WebAssembly plugins for your product, and you discover problems only after they are installed — wrong output formats, unhandled inputs, plugins that exceed limits or call host functions incorrectly. You want a conformance suite authors run before publishing, so plugins that pass it work in the real host.

Prerequisites

  • [ ] A documented plugin interface (exports, imports, data formats, limits) — ideally a WIT world or a clear ABI description.
  • [ ] A host implementation you can embed in a test runner (Wasmtime, Extism, or your own host library).
  • [ ] Representative inputs and expected outputs for the plugin kinds you support.

What the contract covers

A plugin interface is more than function signatures. The contract is everything a plugin must do for the host to work correctly: which exports exist and what they return for each input, which errors are allowed and how they are reported, which host functions may be called and with what arguments, how much memory and time a call may use, and how the plugin behaves across repeated calls (statelessness, idempotency). Much of it cannot be expressed in WIT or a header file; it lives in documentation — and in tests, which make it checkable.

A conformance suite turns the contract into executable checks that load an arbitrary plugin binary, exercise it against standard cases, and report which clauses pass. Authors run it locally and in CI; the registry runs it before accepting a version; your own team runs it when changing the host, to see which existing plugins a change would break.

Layers of a plugin conformance suite Interface checks verify exports and imports match the contract. Behaviour checks run golden input and output cases per plugin kind. Error checks confirm malformed input yields contract-defined errors rather than traps. Resource checks enforce memory and time limits. Host-interaction checks verify calls to host functions through instrumented fakes. interface checks exports/imports, world version golden cases input → expected output error behaviour defined errors, no traps resource limits memory, time per call host interaction fake host functions, call logs

Step 1 — write the contract down precisely

Before writing tests, write the rules: “transform(record) returns a record with the same id; it never removes required fields; it returns error(invalid-input) for records missing id; it completes within 50 ms for records under 64 KB; it may call log at most 10 times per call; it must not retain state between calls”. Each rule becomes one or more tests, and the written version becomes the authors’ reference.

Step 2 — build a harness that loads any plugin

The harness embeds the real host runtime with instrumented host functions:

// conformance/src/main.rs (sketch)
fn main() -> anyhow::Result<()> {
    let path = std::env::args().nth(1).expect("plugin.wasm");
    let engine = Engine::new(Config::new().consume_fuel(true))?;
    let component = Component::from_file(&engine, &path)?;
    let mut results = Vec::new();
    for case in cases::load("cases/transformer")? {
        let mut host = FakeHost::new(&case);              // records log/http calls, serves fixtures
        let mut store = Store::new(&engine, host);
        store.set_fuel(case.fuel_budget)?;
        let plugin = Transformer::instantiate(&mut store, &component, &linker(&engine)?)?;
        let out = plugin.call_transform(&mut store, &case.input);
        results.push(case.check(out, store.data()));     // compares output, errors, host calls, fuel used
    }
    report(&results);
    Ok(())
}

Distribute it as a single binary (and a container image) so authors in any language can run plugin-conformance my_plugin.wasm. Use the same runtime and configuration as production, so passing means something.

Step 3 — golden cases and property checks

Golden cases pair inputs with expected outputs for deterministic plugin kinds. For plugins whose output varies (formatters, enrichers), check properties instead: output parses, required fields are present, id unchanged, size within limits. Include edge cases deliberately — empty input, maximum-size input, Unicode edge cases, missing optional fields — because those are where plugins differ from the author’s happy-path testing.

Running the suite in an author's CI The author builds the plugin, downloads the conformance runner pinned to the target host API version, runs it against the built module, and gets a report of passing and failing contract clauses. Failures block publication; passing results are attached to the release and checked again by the registry. build plugin any language fetch conformance runner pinned to host API run all cases golden + properties report per clause pass / fail + details publish if green registry re-runs

Step 4 — check errors, limits and host calls

Error cases feed malformed inputs and assert contract-defined errors — not traps, not hangs. Resource cases measure fuel or wall time and peak memory per call and fail plugins that exceed limits, with the measured values in the report so authors can see how close they are. Host-interaction cases use fake host functions that record calls: assert that a plugin does not call http when the contract says transforms must be pure, does not exceed log limits, and handles host-function errors (the fake returns failures on purpose) gracefully.

Step 5 — ship the suite with the SDK and gate the registry

Version the suite with the host API: suite 2.1 tests the 2.1 contract. Provide CI templates for common plugin languages that build and run it. Run the same suite in the registry’s publication pipeline; a version that fails cannot be published to the reviewed channel. When you change the host, run the suite against all published plugins to see who is affected before releasing.

Keeping the suite fair and useful

A suite that fails for reasons authors cannot understand gets ignored. Every failure should say which clause failed, the input, the expected and actual results, and a link to the documentation for that clause. Keep timing limits generous enough to pass on modest CI machines, and use fuel or instruction counts where possible for deterministic results. Add a case whenever a real-world plugin bug slips through, so the suite grows with experience.

Testing statelessness and repeated calls

Many host designs reuse a plugin instance across calls for speed, and contracts often require that results do not depend on earlier calls. Plugins break this easily: a global cache that grows without bound, a counter used in output, state left over from a failed call. Add cases that call the same export many times with interleaved inputs and assert each output matches what a fresh instance produces for that input; cases that make a call fail on purpose (malformed input, a host function returning an error) and then check the next call still succeeds; and cases that measure memory after thousands of calls to catch growth. If the contract allows state — a plugin that aggregates across calls — test the documented reset behaviour instead, including what happens when the host discards the instance mid-sequence.

Language-specific templates

Plugin authors use different languages and toolchains, and each has its own pitfalls: Rust plugins that panic on unexpected input, Go plugins that grow memory because of the garbage collector’s behaviour, JavaScript plugins compiled with an embedded engine that start slowly. Provide a minimal template project per supported language that already passes the suite and runs it in CI, so authors start from a working baseline and see failures as soon as their changes introduce them. Templates are also where you document language-specific advice — how to report errors through the contract’s error type, how to keep memory bounded — in the form authors actually read: working code.

Versioned results

Attach the suite’s report to each published version, so administrators can see which contract version and cases a plugin passed, and so you can tell later whether a plugin predates a case added after a bug.

Expected output

plugin-conformance csv_export.wasm runs 64 cases in 3 seconds and reports all clauses passing; a plugin that traps on empty records fails the “malformed input yields invalid-input error” clause with the offending input shown; the registry refuses to publish a version that exceeds the 50 ms budget on the large-record case; and a proposed host change is evaluated against all 120 published plugins before release.

Gotchas

  • Contract only in prose. Authors interpret it differently. Encode it as tests.
  • Harness using a different runtime than production. Passing means little. Use the production host library.
  • Wall-clock limits on slow CI. Flaky failures. Prefer fuel or generous budgets.
  • Opaque failure messages. Authors give up. Report clause, input, expected and actual.
  • Static suites. Real bugs repeat. Add a case for every escaped bug.
  • Only testing fresh instances. Reused instances leak state between calls. Test repeated and interleaved calls.

Performance note

Running the full suite with fuel metering took about 3 s per plugin; across 120 published plugins, a host change could be checked in under 7 minutes on one CI runner.

Conformance suite duration Seconds to run the 64-case conformance suite against one plugin, and minutes for all 120 published plugins expressed in seconds, on one CI runner. seconds one plugin 3 s all 120 plugins 400 s

Frequently Asked Questions

Can authors run the suite without the full host? Yes — the harness embeds the host runtime and fakes for host functions; no product installation needed.

How do I test stateful plugins? Run sequences of calls in one instance and assert the documented state behaviour, plus resets between instances.

Should the suite include performance benchmarks? Include limit checks; leave detailed benchmarking to authors, with optional reporting.

What if a plugin legitimately needs more resources? Make limits part of declared permissions that administrators approve, and test against declared values.

How do I check that a plugin does not depend on earlier calls? Call it repeatedly with interleaved inputs and compare each output with what a fresh instance produces for the same input.

Should authors get starter templates? Yes — one per supported language that already passes the suite in CI gives them a working baseline.

Should reports be published with each version? Yes — attaching the suite report shows which contract version and cases a plugin passed.

← Back to Plugin Systems & Extensibility