Tests can either be defined as functions or as classes with "steps". What's that for?
https://feditest.org/faq/feditest/test-functions-classes-steps/
Consider a Fediverse test such as: “if I reply to the post of my friend, they will see my reply on their system right below their original post.”
Let’s call myself user @me@me.example
on Node me.example
and my friend @you@you.example
on Node you.example
.
To test this, many steps are necessary:
Fediverse Nodes
me.example
andyou.example
need to be set up and put into a pristine state. This may involve manual software installation or a database wipe.Accounts
@me@me.example
and@you@you.example
need to be created. This may involve clicking through a bunch of web pages and waiting for e-mail to arrive.Account
@you@you.example
needs to create a post.Account
@me@me.example
needs to access that post onme.example
.Account
@me@me.example
needs to reply to that post.Only now do we get to the actual test: has the reply has arrived on
you.example
?
Let’s say the test fails: with a typical test framework, the test raises some kind of
AssertionError
, the test framework stops the test run, and, as it is supposed to, cleans
up after itself by deleting all the data created during the test, perhaps even taking
down the Nodes.
This is not optimal for the developer who wants to find out what went wrong. It would be better if, for example, the test run could be stopped right after the failure, and the developer could repeat just the step before the failure. Here: the developer would like to repeat step 5, perhaps more than once, maybe with extra logging or attaching a debugger to a Node, to figure out what’s going wrong.
That’s why in FediTest, we have:
- a flag for interactive mode:
feditest run --interactive
- not just self-contained test functions, but test classes which allow the developer to break a test into as many steps as they like. Steps then can be repeated independently of each other in interactive mode, as in the above scenario, and share state with each other.
So to support this scenario, the developer could define a separate method on their
test class for each step above, annotating them not with @test
but with @step
.
In interactive mode, if step 6 fails, they can repeat step 6 (but not the steps before)
or go back to a previous step (say step 5) and repeat that and so forth.
This would not be possible if all the steps were part of the same test function.
By putting the steps into a class, the steps can also share state. In the above example, step 3 might produce a URI to the generated post, which needs to be given to step 4, so it can access the generated post. As the steps are separate methods (so they can raise exceptions independently of each other), state is shared via instance variables in the test class.