Test Report Guide¶
How to Interpret and Use the Testware Test Report
Introduction¶
This document will describe how to interpret and use the Testware Test Report.
The Test Report is based on the results produced by the automated tests - collecting information from several sources into one user-friendly documentation.
The guide will explain the structure and usability to be able to quickly and efficiently understand the content of the Test Report.
Test Report Structure¶
The Test Report is divided into the levels usually found in a traditional test project.
A Test Report contains a collection of Test Suites - that contains a collection of Test Cases - that contains a collection of Tests.
Following model visualize the structure, names and levels used in the Test Report...
- Test Report Is a document which consists of a summary of all test results executed...
- Test Suite Is a collection which consists of Test Cases grouped within a category...
- Test Case Is a document which consists of a set of actions specifying requirements...
- Test Is a single evaluation of a condition validating the expected result specified...
Summary Page¶
The Test Report summary page contains the overall statistics of the test result. In order to present it in an easily understandable way the report is divided into 6 parts that are described in the following sections.
Sections¶
Properties¶
The Properties section of the Test Report shows application specific details which are used to make sure that the right system of the right version is being tested. The properties are custom to a specific application. The Properties section typically includes:
- Application - The name of an application which is executed and tested
- Company - The name of a company which is operating the application
- Version - The current version of an application that is being tested
- Report Date - The date and time when the report was generated
It is critical to always verify the version number of an application under test, since updates can be deployed at any time. Verifying the report date would avoid situations where an older version of the Test Report is being analyzed.
Statistics¶
The Statistics section provides the user with the overall test run results. This is the first place where the user can see if there are any tests that were not successfully executed. The Statistics section consists of:
- Number of Test Suites - The number of test suites in a test run
- Number of Test Cases - The number of test cases in a test run
- Number of Passed Tests - The number of tests which passed successfully during a test run
- Number of Failed Tests - The number of tests which have failed during a test run
- Number of Skipped Tests - The number of tests which were skipped during a test run
- Number of Inconclusive Tests - The number of inconclusive tests during a test run
- Number of Tests - The number of tests in a test run
The Statistics section summarizes the number of tests that are passed, failed, skipped or inconclusive. Skipped tests are specifically ignored by the test automation team. Inconclusive tests were terminated during the test execution, because of unfulfilled preconditions.
Overview¶
The overview table presents more detailed test run information of different test suites that were executed. Every test suite in the table is provided with the number of overall test cases, number of passed, failed, inconclusive, skipped tests and total tests.
A green Passed label means that all the tests in the test cases in a particular test suite were executed successfully. Even though a test suite may contain skipped tests, it will still be considered as passed. A red Failed label means that some of the tests in the test cases in a particular test suite have failed during execution.
Failed¶
This section contains test cases that failed during a test run. Every test case has its own Id label which is an identification number.
Inconclusive¶
This part of the Test Report summary contains test cases which were ruled out as inconclusive during a test run.
Skipped¶
The last section of the Test Report summary contains test cases which were skipped during a test run.
Test Case Details¶
Test Case detailed view is meant to show all the steps that were executed during a test run of a particular test case. It presents a user with an information about where a failure occurred and which tests have failed.
Test Case Setup¶
- Project – the name of a project
- Id – the identification number of a test case
- Test Suite – the name of a test suite to which a particular test case belongs to
- Name – the name of a test case
- Description – the description of a test case
- Actions – a step which is executed during a test
- Contains time when step was executed
- Contains name of a step
- Contains arguments used in an executed step (Optional)
- Contains a page name where it was navigated to (Optional)
- Message – error message which is shown when a test case fails
- Date – the date and time when a test case was executed
- Status – the end status of a test case after execution is finished
In a case of a failed test the last two lines of an Actions part are telling the user that a screenshot has been saved and also to what kind of file it was saved.
Test Case Image¶
Image presents a particular point in a test case execution process where the test has failed. It can be used to examine the state of the application during the time it have failed.
Sometimes an error occurs outside of the user interface view and cannot be seen in the image.
Test Case Video¶
Video file holds a record of test execution and can be downloaded by clicking on a link. It can help to understand what kind of automated steps are being taken during the test execution. The required software is Windows Media Player.
Tests¶
Test view which represents all the tests that are part of a test case. Test view contains:
- Name – the name of a particular test
- Message – the message which is presented to a user after a test has executed
- Contains the expected result of a test
- Contains the actual result of a test
- Status - the end status of a test after execution is finished