FediTest

2024-09-23

FediTest 0.4 release

https://feditest.org/release-notes/0.4/

FediTest 0.4 has been released. This release is a major improvement over the previous FediTest 0.3 release. Far more things are working, but we also still have a while to go.

To upgrade:

If you have installed FediTest via pip, upgrade with:

% pip install --upgrade feditest

Once you’ve upgraded successfully, you can now check your currently installed version with:

% feditest version

It should report 0.4.

Major new features

  • FediTest now knows how to speak to the Mastodon API. This means we can now run automate both the control and observation of Mastodon Nodes in tests without needing human intervention.

  • Nodes running WordPress (plus the needed plugins: ActivityPub and Enable Mastodon Apps) can also be automated in the same fashion.

  • We have an initial set of system-level Fediverse tests. These system-level tests test functionality such as follow, reply, like, announce and delete. These system-level tests can be run in a fully automated fashion for Constellations whose Nodes run either Mastodon or WordPress + plugins, using UBOS Gears and Constellations such as:

    • Mastodon against Mastodon (e.g. follow and message delivery from one Mastodon instance to another)
    • WordPress + plugins against Mastodon
    • Mastodon against WordPress + plugins
    • Mastodon against Mastodon against Mastodon (e.g. if a post is created on instance A, and boosted from instance B to an account on C, and then deleted, will it also be deleted on instance C?)

  • Locally-running Nodes (i.e. Nodes running in a Linux container or virtual machine) can now communicate with each other over HTTPS at fully-qualified DNS hostnames. This has become possible by running local DNS, and a local certificate authority. To support this with non-public IP addresses in a Linux container, the built-in SSRF protections of WordPress and Mastodon are automatically disabled.

  • As you probably guessed from the bullets above, our focus in the release has been to fully integrate Mastodon with FediTest, as well as WordPress + plugins. This is a good foundation on top of which other apps can be integrated going forward.

  • HTML report generation has been improved: only generate the test matrix if the same tests have been run with multiple Constellations. As a consequence, the Test Plans are now comprised of one session template and one or more Constellations.

  • See some tentative results, check out the current test reports.

Other changes and improvements

  • Instead of having to generate the various The various JSON files used by FediTest one step at a time, you can now specify the tests and the Nodes of the Constellation right on the feditest run command-line. Other subcommands have been similarly made more flexible.

  • The FediTest framework now distinguished between normal Nodes (running apps such as Mastodon) and Diagnostic Nodes which have extra responsibilities and extra abilities to be controlled by tests scripts and report observations. (This is mostly a placeholder for the future, we don’t have code yet that uses this beyond the Imp.)

  • A new scheme for listing already-existing accounts on SaaS-hosted Nodes in the Node definition JSON. This makes it significantly easier to understand and run how to use FediTest with already-running instances.

  • We have an initial set of unit tests for FediTest itself. (How do you test a test framework? Maybe it’s test frameworks all the way down!) We use pyunit for unit tests and feditest itself for smoke tests.

  • The documentation here at feditest.org has been expanded.

  • Lots and lots of other bug fixes and improvements. In doubt, check the commit logs.

Incompatibilities

Some of the JSON files used by FediTest have changed their syntax somewhat. In particular, there is a brand-new way of specifying existing and non-existing accounts in Node definitions, and because the structure of Test Plans has changed a bit, Test Plan JSON files need to be re-generated.

Known issues

  • We are still at 0.x version, so we’re not done yet!

  • ActivityPub messages POSTed by WordPress (plus plugins) do not always arrive in time. That’s because of #397 for which we need guidance from the WordPress / ActivityPub plugin project.

  • Also check the issue trackers.

Next milestone

Expected to be mostly a bugfix release. Labeled “FediTest 0.5 released” on Github.

P.S. Please send bugs, we can’t fix them if we don’t know about them.

Help wanted!

Are you part of a Fediverse software project? Please point them to FediTest, and the Community-contributed test results, in the hope those results will be useful to them to improve their software. And check off their name here.