Software testing automation documentation

Here is what i would propose:

1. Test strategy document:
This outlines the over all testing objectives, what testing goals exists and how is the over all testing will be performed linking all levels from unit test, component test, system test and integration test. This is not a standard or something - but it can be something of this sort.

2. Test suit:
This is the collection of test cases and conditions on when and how each cases needs to be performed. Each set of inputs, procedures and expected output behavior against each elements. There are times when more than just success and failures are noted so that further analysis is done on this.

3. Test environment/setup and procedures
If you are automating the testing process fully or partly, it is worthwhile to document how exactly the (various elements of) testing is going to get executed. It should be debated and validated if the testing performed here is correct method or not. The developer and QA associated should know how to operate the set of tools and what procedures to follow.

4. Traceability Matrix:
This is a well defined matrix which identifies which set of test cases are relevant to promise each functionality point is assured to be steady. Read more here or this wiki link. Whenever a new bug is discovered or a new feature is requested, traceability matrix should be updated to capture these changes.

5. Test results
Whether generated automated or performed manually, the results (detailed and summerised) should be captured in a test execution sheet. Most important thing to note down is that a. original observation (such as logs, actual output of the application) should be capture as relevant so that conclusions can be validate. b. the document needs to capture the build against which these test were carried out; different build may not produce the same behavior against the same test.

Procedure and formats can be developed as needed. Most important thing, from my personal experience is that instead of making water tight compliance to some format allow people do document this like a running diary making only few things mandatory and let people pour in more information freely. Testing is never static (at least for any reasonably complex project) so over time all these template must evolve continuously - quite often every next step could be a major departure from the last one. If either the templates are stale, or people do not follow that because it is too rigid, eventually much of the relevant knowledge through testing procedure won't be reflected right.

The answer greatly depends on what the software does, how it is designed, and even the industry the software will be used in. Some industries have very extensive regulatory requirements for test documentation & validation, for example FDA CFR 21 Part 11in the Pharma and Biotech industries.

Also, consider that even if your organization would religiously follow all the relevant IEEE standards for testing documentation, such as:

• IEEE 829: software test documentation
• IEEE 1008: unit testing
• IEEE 1012: software verification and validation
• IEEE 1028: software inspections
• IEEE 1044: classification of software anomalies

there is a plethora of directly and indirectly related documents and standards you should also define and use, like:

• IEEE 830: system requirements specifications
• IEEE 730: software quality assurance plans
• IEEE 1061: software quality metrics and methodology
• IEEE 12207: software life cycle processes and life cycle data

If all of those IE3 standards look insurmountable - especially for smaller organizations and teams - have a look at the Software Engineering Body of Knowledge which is less "scientific" and more concise.

Finally, to some extent, using ALM tools that keep electronic records of tests in stead of formal documents is a perfectly valid alternative as long as you have procedures that define a standardized use.

In my project, the automation tests are essentially documented using the same means as manual tests. In addition to all the other information, we also point to the test case in code.

We maintain an automation suite which the whole team knows how to execute. With both these pieces of information, any member on the team is able to identify the test case as well as run it.

Also, like our production code, we also version and review the automated test cases. These serve as living documentation for the code representation of the test.

So in a way, we are reusing our documentation ways for manual tests and production code for automated tests.

Did you look at IEEE 829-1998, Standard for Software Test Documentation? I know that in my company the test documents in this standard are used to describe the entire testing strategy. For example, the Master Test Plan will actually include information on testing automation in the approach section.