Is this feature ready to document? Using AI to assess source information

A feature can be ready for release and still be impossible to document properly.

Let me explain.

The technical writer might receive a completed development ticket, a few screenshots, and a message saying, “It works just like the previous version.”

What is missing from that is the information needed to explain the feature accurately. Who is it for? What problem does it solve? What permissions are required? What happens when something goes wrong? Has anyone confirmed the final behaviour?

The writer can investigate those questions. However, that work takes time and makes documentation schedules difficult to predict. It creates “grit”, friction, in the technical writing process, right from the start.

Cherryleaf has been working on an approach where our clients can use an AI-generated documentation readiness score to identify these gaps before the writing starts.

What does documentation readiness mean?

Documentation readiness is a measure of whether a technical author / technical writer has enough reliable source information to begin or complete the documentation.

It is not a measure of whether the software is ready to deploy. It does not assess code quality, test coverage, or operational readiness.

It assesses the information available to the writer.

Depending on how the software is developed, the AI can examine:

• Development tickets and acceptance criteria
• Product specifications and architectural decision records
• Pull requests and code comments
• API specifications
• Test cases
• Meeting transcripts
• Screenshots and working builds
• Existing documentation

It then compares the available evidence with an agreed rubric.

The result is a number, plus an explanation of which information is missing, where statements conflict, and what would increase the score.

What problem is the score trying to solve?

Technical writers often discover information gaps after work has been scheduled.

For example:

• A ticket might say the feature is complete, but not identify the intended user.
• A specification might describe the expected workflow, but the working build behaves differently.
• Error messages, permissions, and limitations might be absent.

In that situation, the writer then has four choices:

• Start writing and mark up large parts of the draft for review.
• Spend time interviewing developers and product managers.
• Make assumptions, with the risk they could appear in the published documentation.
• Stop the work (and explain why it cannot proceed). This option often leads to a time crunch at the end of the project.

A readiness score moves that conversation earlier, in a similar way to Lean methods that highlight and fix problems.

It gives the writer a quick way to triage incoming work. It also gives the development team specific feedback, such as:

The source pack scored 62 out of 100. The main workflow is documented, but the permission model and failure behaviour are missing. Add the relevant test cases and confirm the required user role to reach the writing threshold.

That is more useful than “we need more information”.

A possible scoring model

A useful rubric might contain six categories, with each category having defined scoring levels. For example:

• 0 means no relevant information was found.
• 1 means the source contains an unsupported assertion.
• 2 means partial information exists, with important gaps.
• 3 means the information is sufficient and supported by evidence.
• 4 means the information has been verified against the implemented product.

The weighted result produces a score out of 100.

The thresholds will vary between organisations, but one starting point is:

• 85 to 100: Ready to document and review.
• 70 to 84: Writing can begin, but named gaps must be resolved before publication.
• 50 to 69: Discovery work is required before drafting.
• Below 50: Return the work for more source information.

We recommend the bands are tested against real projects, as there isn’t a universal standard.

Keep blockers separate from the total score

An average can hide a serious omission.

For example: A source pack might contain detailed UI labels, examples and test results. It could still omit the permissions needed to perform the task.

Certain gaps should, therefore, override the numerical score. Possible blockers include:

• The target user or use case is unknown.
• Sources disagree about the product’s behaviour.
• Security, safety, or compliance information has not been approved.
• The writer cannot verify the final UI, API, or command-line, behaviour.
• A product decision affecting the documentation remains open.

There could be situations where a pack with a score of 88 and an unresolved security question is not ready for publication.

This means a dashboard should show both results:

Readiness score: 88
Status: Blocked
Reason: Permission rules have not been approved.

What AI adds

A technical writer could complete the assessment manually.

AI can add value when there are many tickets or source files to assess.

An AI agent or app can find any absent sections, compare statements across sources, and turn unstructured notes into a consistent report. It can also reassess the material when a developer adds new evidence.

Output

Regardless of whether the assessment is carried out by a human or machine, the output should contain:

• The overall score and category scores
• A status and any blocking conditions
• Evidence supporting each score
• Missing or conflicting information
• The confidence of the assessment
• Specific actions that would increase the score
• The person or role best placed to provide each answer

The evidence justifying the score matters more than the score itself.

How to make the score fair

A score used inside a workflow needs to be consistent enough for people to trust it.

Start by scoring the source material, not the developer

The result describes the state of an information pack at a particular time. It is not a judgement of someone’s ability or effort.

Every score should point to evidence

If the AI awards points for documented error behaviour, it should link to the test case, specification or transcript containing that information.

Missing evidence should be labelled as missing. It should not be reported as evidence that the feature lacks a capability.

Calibrate

Teams also need a calibration exercise. Give the same set of completed source packs to technical writers and the AI. Compare the results and discuss the differences. Rewrite any scoring rule that produces inconsistent interpretations.

Record the model, prompt, and rubric version used for each assessment

A score produced in March should remain understandable after the model or rubric changes in June.

Allow a human override

The technical author might know that a low-risk wording change needs less evidence than a new authentication workflow. The reason for the override should be recorded.

Five ways to use the score

The most immediate use is triage. Writers can identify which projects are ready to start and which need more discovery.

From there, it depends on your relationship with the development team.

The second use is developer feedback. Instead of sending a general request for more detail, the writer can return a short list of missing facts.

The score can also support recognition. You can use “social proof” to encourage others. Teams that provide clear, verifiable source information should receive credit. It could be cake or a certificate. Or it could be an all-company email thanking the team.

Share examples of good source packs and explain what made them useful. Recognition for improvement is often fairer than a league table based on raw scores.

A readiness threshold can become part of the definition of done. For example, a feature cannot move to “documentation ready” until it reaches 80%, has no blockers, and names an owner for remaining questions.

Senior managers can use aggregated scores to find process problems. Useful measures include:

• The percentage of source packs that meet the writing threshold
• The most common blocking categories
• The median score when documentation work is requested
• The time taken to resolve information gaps
• The difference between the initial and final assessment

Be careful when turning the score into a KPI

Canonical’s documentation methodology provides a useful comparison. It introduced shared standards, developer education, and quarterly documentation objectives. Progress was tracked across teams, with a strong emphasis on making good practice easier. The account published by DX describes the approach, while Canonical describes documentation as a shared engineering responsibility.

There is also a warning in the experience. Some teams reportedly used documentation scores as developer KPIs, then stopped because the assessments were considered too subjective.

That risk does not disappear because AI produces the number.

Once a score affects performance reviews, people will optimise for the rubric. A readiness score is safer as a process KPI than a performance KPI. Use it to ask why source information arrives late, which questions recur, and where ownership is unclear. Do not use it to rank individual developers.

The score should start a conversation

A documentation readiness score will not remove the need for technical writers to investigate, test, and question the product.

It can remove some avoidable work. It can expose missing information while developers still have the details in mind. It can also replace vague disagreements with a list of evidence that is present, absent, or unresolved.

The goal is to make sure the writer has something reliable to work with.

Need help with using AI in technical documentation?

Cherryleaf can help. See:

• Cherryleaf’s AI-related documentation services
• Using Generative AI in technical writing | ISTC accredited course
• Managing and mastering documentation projects with AI