FediTest first draft WebFinger results (for FediDevs meetup today)

For today’s meetup, here are some materials.

As a reminder:

• FediTest user stories
• FediTest architecture

We can report on the first back of testing for the simplest part of this all: WebFinger service conformance.

• Annost-annotated RFC 7033 (WebFinger): this is where we identified what to test.
• Test results as test matrix: provides an overview over 17 tests against 39 implementations.
• Test results as test transcript: more detail, one implementation at a time.

There’s also:

• Test results as experimental TAP report: the TAP test format is a bit out of its depth here, we may not keep generating this. Feedback appreciated.
• The FediTest-native report in JSON: the other reports are generated from this one.

We are looking for feedback on everything, but particularly:

• Are results like this useful to you? How can we make them better/more useful?

• How should we report discrepancies between what the standard clearly says, what the standard probably means to say but isn’t entirely clear about, and what is widely implemented? Examples:

  • application/json is widely used as the JRD content type, not application/jrd+json as the spec implies. Error? Not error? Hard error vs soft error?

  • Some HTTP status codes other than what the spec says. E.g. some implementations report 422 or 400 when attempting to resolve an unknown URI scheme. How should we report that?

Best avenue for feedback: file an issue on framework and reporting or file an issue on the actual tests. Also: Matrix group.

Update 2024-06-07: Recording of the (most of the) call at: https://diode.zone/w/j2fXb58GYMe6n6iFJZYGYs