Docs as tests: documentation that can prove it still works

“We’ve long thought of documentation as something to be repaired. The testing process quietly assumes something else, which is that documentation is something to be executed.”

I find that quote from Stephen Mallette significant because it highlights an important distinction. AI is making this issue even more important.

Some organisations end up treating documentation maintenance as a repair job. A customer, support agent, or Technical Author discovers the mistake, and then the documentation is corrected. It’s often called “documentation drift” because the product and the instructions have drifted apart.

Docs as tests offers a different model. It turns documented procedures into tests that can be run against the product. Instead of waiting for somebody to spot an error, the organisation checks whether the instructions still work.

Documentation becomes part of the product’s quality system.

What are docs as tests?

Docs as tests is a strategy for checking documentation against the product it describes.

A documented procedure might tell a user to:

1. Open a particular page.
2. Select an option.
3. Enter some information.
4. Click Create account.
5. Confirm that a dashboard appears.

Each instruction contains a claim about the product:

• The page exists.
• The option is available.
• The button has that label.
• The expected result appears.

A documentation test turns those claims into executable actions. You can use software to follow the instructions in a browser or command-line environment, or against an API, test the claims, and report whether the product behaved as documented.

The Docs as Tests project describes the approach as a way to validate guides, tutorials and API references before users encounter a problem.

It works in both directions. A failed test might reveal an outdated guide. It might also reveal that a product update has broken a working process.

AI has changed the case for documentation

Software companies have asked for years whether they still need documentation.

AI has given them a fairly blunt answer: yes.

AI assistants, retrieval systems and agents depend on source material. If the help centre contains an obsolete procedure, an AI assistant can repeat it to thousands of users. If an AI agent acts on those instructions, it might attempt the obsolete procedure itself.

The scale changes the risk.

A person reading a poor guide might recognise that something is wrong and try another route. An AI system might reproduce the error consistently, confidently, and at speed.

Better language models do not solve this problem. The model still needs reliable information about your product, policies, and procedures.

Testing documentation therefore supports two audiences: it helps the person following the guide, and it improves the material available to AI systems.

There is some evidence that the effect is measurable. Kong reported that its documentation chatbot’s answer accuracy increased from 84% to 91% after its CLI how-to guides were made testable. The improvement came from the documentation rather than a change of model, according to the Docs as Tests account of the project.

From maintenance to verification

Traditional documentation maintenance depends heavily on events outside the documentation team.

For example:

• A developer remembers to tell the technical writer about a UI change.
• A customer reports a discrepancy via a support ticket.
• The development team includes documentation in the release plan.

All of these rely on people remembering the documentation should be updated.

An executable test provides evidence. The procedure passed, failed, or could not be run. A failure can point to the step where the product and the documentation parted company.

Documentation tests can also run in a continuous integration pipeline. A change to the product or its documentation can trigger the relevant tests. The team finds the discrepancy during development, rather than after publication.

AI can help write the tests

Writing documentation tests by hand takes time. Every new procedure adds a potential test. Large language models offer a practical way to reduce the setup work. An LLM can read a procedure and convert its steps into a structured test specification. AI should not decide whether the test passes each time it runs.

A typical workflow is:

1. The LLM reads the documented procedure.
2. It converts the instructions into test actions.
3. A validator checks that the test specification has the correct structure.
4. A person reviews the proposed test.
5. A deterministic test runner executes it against the product.
6. Failures are traced back to the relevant documentation steps.

Manny Silva describes this as “generate once, run forever” in his article on AI-assisted test generation.

Let’s talk about authentication

Public pages and unauthenticated workflows are the easiest place to start.

Authenticated applications require more planning, because they require users to provide a username, password, and perhaps a multi factor authentication code.

A tester could sign in manually before a local test, but that is unsuitable for regular automated testing.

Saved sessions and cookies can work for local runs, but they become unreliable when sessions expire.

For scheduled or continuous testing, a safer arrangement normally includes:

• A separate test environment.
• A restricted test identity.
• Representative but non-sensitive data.
• An approved test runner.
• A secure way to create short-lived authenticated sessions.
• Restricted access to reports, screenshots and recordings.

The test should never gain more access than the documented procedure needs.

What documentation tests cannot prove

A test can confirm that something works as described. It cannot:

• Prove that an explanation is easy to understand.
• Decide whether an overview gives a new user the right mental model.
• Confirm that a warning communicates the seriousness of a risk.
• Tell whether the information architecture helps someone find the correct guide.

Documentation testing complements editorial review, usability testing and subject-matter review. It does not replace them.

Passing is evidence, not absolution.

Start with one important procedure

Start with one procedure that is valuable, frequently used and reasonably safe to automate. Good candidates include account setup in a test environment, a common API request, or a command-line installation check.

Generate the test. Review it. Run it. Observe what fails and how useful the resulting report is.

Then decide where testing provides the greatest return. High-risk procedures, common support journeys, and instructions used by AI agents are sensible priorities.

If you need help, Cherryleaf can set this up for you.

Docs as tests changes documentation maintenance from a largely reactive activity into a repeatable check. It also creates a clearer relationship between the documentation, the product, and the people responsible for both.

The documentation has to tell the truth. Increasingly, it also has to prove it.