FediTest implementation update

We are making progress! Here is where we are:

• The FediTest framework now has the right structure. (I think …) Here are the core abstractions:

  • TestPlan, consisting of one or more TestPlanSessions, each of which is a sequence of tests that are run against a Constellation of Nodes. TestPlans can be generated automatically and edited manually for maximum flexibility.

  • A Constellation of Nodes is essentially a simulated, small Fediverse consisting of two or more Nodes running software and attempting to communicate with each other. For each test session, a Constellation is set up, the tests are run, and the Constellation is taken back down.

  • A Node, you guessed it, is a website that runs a particular Fediverse application. The Fediverse application running at a Node can be unmodified as it would be deployed into production, or instrumented with extra APIs for better test automation.

  • The Node is managed (application, DNS etc provisioned, unprovisioned etc) by a NodeDriver, whose responsibility it is to know how to do this. We have a few NodeDrivers so far, including the ManualNodeDriver (the user has to do everything based on prompts), the SaasNodeDriver (degenerate version for software that only exists as SaaS) and the UbosNodeDriver, which uses UBOS Gears for management, as it knows how to do install and restore complex server-side applications and we can reuse this. (Note: UBOS is one of “my” projects.)

  • Subtypes of Node let us instrument the running Nodes, by making them “do things” (“now post a note!”) and report on what happened (“has the private message arrived?”). We do not have implementations of this functionality yet, except for for certain limited fallbacks. (See next steps below.)

  • TestRun, which is the instance of a TestPlan, and which produces results either in an ad-hoc format or in the TAP format.

  • Tests can come in two flavors: (1) single functions marked as @test run from the beginning to the end and may succeed or fail; (2) classes marked as @test run several member functions annotated with @step in sequence while maintaining state between the steps. The second one is much more manageable for the complex setups many of the more interesting tests require.

  I believe the FediTest framework now meets all the requirements for being handle the types of tests we want to run that will get us to good test coverage over time. (It was not easy! I will probably need to eat my words :-P )

• We have categorized tests into multiple categories for better management. See separate directories in the tests directory of the FediTest-tests-Fediverse repo.

  • Some of the categories match specific standards documents, such as ActivityPub and WebFinger etc. What to test based on the respective standard is annotated with Annost and the annotations are found in the respective TEST-SOURCES subdirectory. To look at them, you need to (1) check out the repo, (2) run git submodule init (3) run git submodule update and (3) open the annotated HTML file in the browser.

  • In lieu of having official standards, we’ve done the same for non-standard SWICG documents, such as the recent “ActivityPub and HTTP Signatures” community report.

  • The fediverse directory tries to define how it is all put together into high-level tests that approximate what users might expect. Weirdly enough, there aren’t any documents that describe what those expectations are known to be, or should be. So we collected user scenarios from the community.

• There are some actual tests in those directories, but not many yet. Now that we have the right framework in which to express them, and we have identified what the documents say should be tested, it should be relatively straightforward to write them down in Python.

  Some examples:

  • One test for a WebFinger server: Do not accept malformed resource parameters. Note the name of the test file is based on the section in the Annost-annotated WebFinger RFC where the test was found.

  • One test for a WebFinger client: Accept unknown entries.

  • One test for full-fledged Fediverse interactions: Follow.

If you want to try and run some of this, go to Quickstart.

Next steps:

• Code more Fediverse tests similar to Follow using high-level APIs, such as the one we’ve started on FediverseNode. Example method: make_a_follow_b.

• As we code more high-level tests, expand the high-level APIs.

• Create a MastodonClientNode implementation that maps those APIs to the Mastodon client API, which has been implemented not just by Mastodon but by several other Fediverse applications. We expect that we will be able to automate many tests by mapping to this API.

• I’d really like to see a gRPC mapping, which we could use as a language-agnostic way of driving Fediverse applications that aren’t written in Python and aren’t using the Mastodon API.

• Start producing actual reports from actual testing.

This is all work in progress, of course, so YMMV! Issues? Questions? Chat with us on Matrix on #fediverse-testing:matrix.org.