Test runs

What is a test run in software testing?

A Test Run is a single instance of executing a specific set of test cases.

A Test Run may consist one Test case, a bunch of them, whole sets of Test cases (Test Suite), or even Test cases from different areas bundled together in a Test Plan.

There are two ways to start a Test Run: 1. Express Run - from the Project Repository page. 2. Regular Run - from Test Runs page.

Express Run

You can quickly set up a Test Run for one or more test cases directly from the Repository.

Hit the "Run" button and proceed to select the Environment, Milestone, Configurations, or any other fields, as needed.

Regular test run

To create a regular test run, navigate to "Test Runs" section, and hit the "Start New Test Run" button. You will see the same modal window as seen while creating a Test Run from the Repository page.

Run Properties

Run title: it is automatically set to the current date. You can replace it with your preferred title.
- Eg: "Regression Test - Release 2.0"
Description: give additional details about the test run.
- Eg: "This test run verifies new features and bug fixes in the latest release.
Environment: define in which environment the test run should be performed.
- Eg: testing, staging, production.
Milestone: select which Milestone tied to the Test Run.
- Eg: Release 2.0
Configuration: choose from pre-defined configuration options.
- Eg: OS: Windows ; Browser: Chrome
Custom Fields: set custom field values if previously configured.

Choosing Test cases: Here, you'll be presented with three options.

From Respository.

From Test Plan: You can choose all or select Test cases from a Test Plan.

From Saved Queries: You can read more about Saved Queries here.

NB: Queries are available with Business and Enterprise subscriptions.

Test runs view

Here, you can look at all of your Test runs, with their Author, Environment, Time Spent and Status with a visual summary of their results.

You can filter the Runs by Status, Assigned to, Environment, Milestone, Tags, Automation status, Creation date, and your custom fields (if any).

Dashboard

To go to the Run dashboard, simply click on the Test Run name.

Let's explore the options available in the dashboard -

1. Open Wizard: this will guide you through the Test Cases contained in the run, step-by-step.

2. Share Report: turn on public link and easily share your Run report with anyone, even if they don't have an account in Qase.

Example of a Shared report

3. Export: you can download a CSV export of your Test Run.

4. (---) Menu: Clone run option allows you to create a new identical copy of your run. You can edit the run details, abort it or go the Test Run settings from here.

5. Search and Filters: search cases by their title or Id and filter by a specific parameter.

6. Defects and Team stats: view all associated defects from this run and see stats for results by assignee.

7. The "..." menu button: has options to Run wizard, assign case to a team member or View/edit the Test case itself.

8. Run details: this side bar houses a completion chart and other test run configuration details like -

Environment, Configurations, Milestones and tags
Run status, author, create date and estimated time for completion.
Linked External Issues.

Expand to learn more

Donut Graph shows the breakdown of Test Cases in the Run by status. Hovering over a section of the graph you will see the actual number of Cases in a respective Status. If you click on a section of the graph, a quick filter will be applied to show you only the Cases in a specific status.
Completion Rate represents the percentage of Cases in a run that have been performed so far.
Status: This lets you know if the run is currently active or completed.
Started by and Started at - provide details on who started this Test Run, and when.
Estimation: It predicts how long your current Test Run will take based on the time spent in previous Test Runs on the same Test cases.
Duration: the summary of time that has been spent on all Test Cases in the current Test Run so far.
Environment, Tags, Configurations, Milestones, and Custom Fields: Details about the specific environment of the test run and the associated tags, Milestones and Custom fields.
External Issue: You can link a test run with an issue/task from any issue tracker Apps that you have installed.
Here's an example of a JIRA issue linked to a test run:

9. Bulk action: You can select multiple test cases and click on 'Update selected' to retest / submit result / manage assignee / remove test case from run.

Example

Defects Tab:

The Defects tab provides information about failed test cases in the run, including the reporter, assignee, third-party app connections, and status.

Team Stats Tab:

The Team Stats tab provides information about your teammates in the Test Run, including their work time, results, and a performance trend chart.

Test Run Wizard

Opening is the wizard is easy, just click on a test case in a Run. You can also get to the wizard in the following ways:

Click "Open Wizard" in the Dashboard:
Click 'Open Wizard' in the Test Runs Menu
Click "Run Wizard" in the "..." menu of a Test Case:

In the wizard, can you advance through Test Cases, add comments, attachments, and log results for both individual steps and the entire Test Case.

You can also file defects as you work through the test case. Check the defects article for more information.

In the wizard, use the "View/Edit Case" buttons to open a test case in a new tab for viewing or making changes.

NB: If you've enabled the "Fail case on step fail" option in Project settings, failing any Test Case step will automatically result in the entire Test Case failing. To prevent this, adjust the setting.

Editing a completed Test Run

After finishing a Test Run, you can add results by enabling "Allow to add results for cases in closed runs" in settings. Then, go to Test Runs, open the run using the "Open Wizard" option. In the wizard, click the edit icon (pencil) to adjust the run duration, add comments, and attachments.

Save changes with the green check mark or discard them with the red cross.

Test Run Settings

Under Project Settings, there's a dedicated section for modifying run behaviour.

Let's look at each option in more detail:

Option

Behaviour

Fast Pass

Enabled: you won't be prompted to add extra details when marking a test case as "passed"/ "skipped."

Disabled: A modal window appears for adding extra details like comments or attachments when marking a case as "passed"/ "skipped."

Default create/attach defect checkbox

Enabled: The checkbox for creating a defect is checked by default when you fail, invalidate, or mark a case as "blocked."

Disabled: The checkbox is unchecked by default in those scenarios.

Auto complete

Enabled: A run is automatically marked as complete when all cases have a result.

Disabled: You need to manually mark the run as complete, even when all cases have a result.

Auto passed

Enabled: Marking all steps of a case as "passed" automatically marks the entire case as "passed."

Disabled: You need to separately mark the case as "passed," even if all steps are "passed."

Auto assignee

Enabled: Unassigned test cases are automatically assigned to the first person who opens them in the Wizard.

Disabled: Unassigned cases remain unassigned until explicitly assigned.

Auto create test cases

For results reported via API or an API reporter.

Enabled: If the result refers to a test case that is not in your Qase repository, a new case is created. Also, any updates will be made
Later on, if something changes in that test case upon the next result submission, related test case in the repository will get updated accordingly with this option enabled