End-to-End Skill Testing with OvoScope

A skill that "probably works" is not tested

Most OVOS skill repos ship without automated tests. The ones that try usually mock the message bus, so the test never runs the real intent pipeline, the part most likely to break. Anything past "did it import" was checked by hand: a real device, a microphone, and someone reading logs to guess whether the right thing happened.

OvoScope closes that gap. It runs a complete OVOS Core pipeline in-process and lets you assert on exactly what the message bus produced. No hardware, no server, no guesswork.

What it checks

OvoScope loads real skill plugins onto a FakeBus, an in-memory stand-in for the OVOS MessageBus, and drives a real IntentService and SkillManager. You send a test utterance as a Message. OvoScope captures every message that comes back, so you can assert on message type, data fields, routing context, session state, and message order.

from ovoscope import End2EndTest
from ovos_bus_client.message import Message
from ovos_bus_client.session import Session

session = Session("test-session")
session.pipeline = ["ovos-adapt-pipeline-plugin-high"]

utterance = Message(
    "recognizer_loop:utterance",
    {"utterances": ["what time is it"], "lang": "en-US"},
    {"session": session.serialize(), "source": "A", "destination": "B"},
)

End2EndTest(
    skill_ids=["ovos-skill-date-time.openvoiceos"],
    source_message=utterance,
    expected_messages=[
        utterance,
        Message("ovos-skill-date-time.openvoiceos:TimeIntent",
                data={"utterance": "what time is it"}),
        Message("speak"),
    ],
).execute()

Pass that test and you know the real pipeline (real Adapt or Padatious models, the real skill handler, real session context) produced the expected intent and spoke a response, in that order.

End2EndTest has a set of knobs, and each one targets a real failure mode:

eof_msgs / eof_count mark when a dispatch lifecycle is finished (the default terminator is ovos.utterance.handled).
ignore_messages / ignore_gui drop noise you are not asserting on.
flip_points, entry_points, and keep_original_src check that source/destination routing flips correctly as a message travels through the stack.
activation_points / deactivation_points assert which skill is active after a given message, the mechanism behind conversational follow-ups.
skill_id / pipeline_id isolate a single lifecycle when two dispatches interleave non-deterministically, such as a stop request landing mid-response.

Each sub-check (test_msg_data, test_routing, test_active_skills, and others) can be toggled off individually when a test only needs to cover one dimension.

The harness underneath

End2EndTest runs on MiniCroft, a trimmed SkillManager subclass that handles config isolation, skill loading, pipeline initialization, and teardown. OvoScope ships it as a class-scoped pytest fixture, auto-discovered through the pytest11 entry point. Installing the package makes the fixture available, with no plugin to register by hand.

Not every skill reacts to an utterance. Timers fire, GUI buttons get pressed, other services publish events. For those, MiniCroft.inject_message() puts an arbitrary Message straight onto the FakeBus, so you can trigger a non-utterance handler and assert on whatever it emits.

The optional ovoscope[pydantic] extra bridges to ovos-pydantic-models for typed, schema-validated access to message fields, instead of raw dictionaries.

MiniCroft covers the skill pipeline, but it is not the only harness in the box:

MiniPHAL tests PHAL plugins (the hardware-abstraction layer, which talks purely over the bus) without real hardware.
Separate harnesses, behind their own extras, cover the audio, media/OCP, and listener stacks.

Every harness shares the same shape: a FakeBus, real components, and assertions on real messages.

Recording fixtures instead of writing them

Hand-writing the expected message sequence for a complex flow is tedious and error-prone. The ovoscope CLI records it instead. ovoscope record spins up MiniCroft in-process, sends your utterance, captures the full response sequence, and saves it as a JSON fixture:

ovoscope record --skill-id ovos-skill-hello-world.openvoiceos \
    --utterance "hello" --output fixture.json

Pass --live and it records against an already-running OVOS instance over the real MessageBus instead. Either way, the fixture becomes the expected sequence:

ovoscope run fixture.json replays it and exits non-zero on any mismatch.
ovoscope diff expected.json actual.json shows exactly what changed, with colored output.
ovoscope validate schema-checks a fixture.

Change a skill, re-run, and see exactly which messages moved.

There is also ovoscope coverage, which scans a workspace root and reports which skills have end-to-end tests and which do not. It turns "we should test the skills" into a concrete, trackable list. The full command set, from ovoscope --help on version 1.8.4a1:

usage: ovoscope [-h] COMMAND ...

End-to-end test framework for OpenVoiceOS skills.

positional arguments:
  COMMAND
    record        Record a fixture file.
    run           Replay a fixture and exit 1 on failure.
    diff          Compare two fixture files.
    validate      Schema-validate fixture files.
    coverage      Scan workspace for E2E test coverage.
    bus-coverage  Run fixture files and report bus handler/emitter coverage.

What the sweep found

OvoScope's own test suite covers the framework itself: the End2EndTest assertions, the CLI, the diff engine, the PHAL, audio, media and listener harnesses, and the coverage tracker.

The more interesting result is what happened when the framework was pointed at the skills. The default OVOS skills went through a cross-repo end-to-end sweep built on OvoScope, one suite per skill repository. Per skill, the sweep checked:

Intent coverage: does every documented utterance variant reach the intended intent?
Response sequence: does each handler emit the correct bus messages, in the correct order?
Multi-turn context: do conversational skills track the active skill across turns?
Session isolation: do separate sessions keep independent state?

Because the tests run the real pipeline, they surfaced real bugs: intent slots matching partial phrases they shouldn't, responses firing in the wrong order on certain runtime configurations, and context-handling edge cases in multi-turn skills. Each finding got a targeted fix, and each fix landed with a test that pins the behavior down.

Try it

Install OvoScope and it registers itself as a pytest fixture, with nothing to wire up by hand:

pip install ovoscope

Write an End2EndTest against one of your skill's intents, or record one from a running instance with ovoscope record. Either way, you get a fixture you can replay in CI: ovoscope run fixture.json fails the build the moment a skill's message sequence changes.

That is the bar this sets for 1.0: a skill that ships without a passing OvoScope suite is unverified. Because OvoScope is on PyPI and installs as a pytest plugin, any skill repo can run the same suite in CI, and SpecMessage checks the captured messages against the OVOS message spec. A passing suite is real evidence, not a mock agreeing with itself.

Limits

OvoScope tests the bus contract, not audio. Wake word, STT and TTS quality are out of its scope; the listener and audio harnesses drive those services with pre-recorded or synthetic input. A suite is only as good as the utterances it lists, so a skill with one recorded fixture per intent is covered thinly. Bugs and questions go to the issue tracker.

This work is part of the OpenVoiceOS From Beta to Breakthrough milestone, funded through the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101135429. Additional funding is made available by the Swiss State Secretariat for Education, Research and Innovation (SERI).

Help Us Build Voice for Everyone

OpenVoiceOS is more than software, it's a mission. If you believe voice assistants should be open, inclusive, and user-controlled, here's how you can help:

💸 Donate: Help us fund development, infrastructure, and legal protection.
📣 Contribute Open Data: Share voice samples and transcriptions under open licenses.
🌍 Translate: Help make OVOS accessible in every language.

We're not building this for profit. We're building it for people. With your support, we can keep voice tech transparent, private, and community-owned.

👉 Support the project here