Introduction to TestRail

 

TestRail is a powerful test development tool used by testers and managers all over the world. By choosing testrail, you unlock a key ingredient to help you prepare your own recipe for success. Using testrail, you can document test cases, plans, and runs, as well as capturing real-time results and generating meaningful reports.

Testrail can be accessed through any web browser. TestRail also has full software maintenance and automatic updates. The dashboard provides an overview of projects and recent to-do activities. Todos are on the right-side bar. You always create different projects on each actual product. There are the overview, to-dos, and project activities. You always want to create different products for each actual product but add the multiple versions of a product onto the same project.

The Top of the Screen has overview, to-do, milestones, test runs and results, test cases, and reports. To mark a project as completed, click edit and add a checkmark to the project is completed field. This moves the project into a separate section on the dashboard. Now with an overview, now we can begin organizing projects in TestRail.

 

Now we will learn about TestRail’s different project types, and projects are very important in testrail, and we a can choose between a single repository for all cases, a single repository with baseline support, and multiple test suites to manage cases.

A single test suite is easy to manage and flexible enough for most projects. You can further divide the projects into sections and subsections and using a single repo allows for full end-to-end testing within a single run, while also maintaining concurrent testing with versioning using milestones and test plans.

 We can also use baselines to manage multiple versions of the test cases at the same time. This allows multiple product releases to happen in parallel. After this, you create a master set of test cases and split them into many different baselines, allowing to make changes to the test cases with the baseline without allowing for the master test cases.

The type using multiple test suites to manage cases was still available but should be required for teams requiring much stricter divisions between different testing areas in their system. Multiple suites splits the project into many branches and testing areas. There are limitations creating runs when the scope is beyond the cases included in a single suite. Users can also change a project type.

Now we will see how to create, manage, and organize test cases inside testrail. Test cases are an essential aspect of product testing. To begin with, we can select the “test cases” tab and group these test cases together. Testrail helps solve this grouping issue by giving the option of adding test sections.

Grouping test cases by application or functional area models works best. We have test cases grouped into sections and we can also add subsections. TestRail makes it easy to add sections without the need to navigate to different pages. TestRail also supports drag and drop, so you can rearrange things easily with your mouse. We can cope move cases between projects and suites via the copy and move dialog button. Simply check the boxes of the areas you want to cope and move and then select the “copy” button at the bottom. This is to help prevent accidently moving tests from the source project.

Test cases have various attributes: Title for a name, section to determine which section in the project, template to determine what fields should be present in the test cases, test type to define what manner of test you will be creating, the priority/importance of the individual test, estimate for a general guideline in how long the test will take to execute and the test steps and expected results fields.

You can also have rich text fields to include certain links and more attributes to the test. Testrail as a result uses a modified version of the formatting system, called markdown. For example by adding 2 asterisk characters you can make it marked down. Another thing you can do is configure test cases with individual steps, entering test steps or expected result, or manage test steps separately in a more structured way. We get expected results and also record the results for each step during execution.

We can see all the results associated with a test case at a glance. We can also click on  the tests and results tab and there is also a defects tab which allows viewing of issues from an integrated issue tracker. Finally, you can click on the history on the sidebar and see the full test case history with all the revisions. Testrail offers csv and XML import options as well in how to import test cases.

The next element that we want to discuss in terms of the testrail are test runs and results. There are test runs currently active for a particular project and the application allows us to start multiple test runs using the same test cases over time. You can even create multiple test runs for many different operating systems.

Here, we can provide the title of the test run and the user assigned to it, as well as assigning the test run to a milestone. Test rail includes all the test cases in the project, but we can choose a subset of the test cases, we can manually select cases, or even entire sections. We should have test rail include the most important test cases based on the priority feature.

Another useful option is to select the cases based on their type and combine multiple filters and combine multiple test fields. Once the test run has been created you can go over the test run pages, and go through the status runs over all the individual tests. A tester would usually select a test and open it to verify the test and its results. As you can see all of the test case details are on the following page.

Once a test has been performed, you can add a test result. You can choose between various statuses and testrail allows to customize own statuses in an area, and can add additional error messages. You can assign it to another tester or even a team lead who can look at the test. We can also record and enter the elapsed time we can take to execute the test, and test rail can use this to improve the time forecast and estimates. The defects field is used to link test results into an integrated defect tracker, such as Jira. If a test is failed, you can change the status, try to retest it, or try to test someone else. Afterwards, everything is displayed at the bottom of the page.

When you go back to the test run overview page, you can see that the status of the test case and the test run have been updated. You can also use sorting and grouping options to group tests by status by clocking on the header.

There is also a method to customize display columns in terms of viewing and displaying the test results and adding all of them on the header. You can also generate print report or export, closing the run to protect them from further modifications. Due to interactive user interface, you can change the status of the test case. 

The next video mainly discusses defect integration. We will learn all about the integration in testrail as well as various debugging tools. We should do the configuration process, which doesn't take too long. First we head to administration:

Then Integration. We use the Jira Integration wizard for this scenario which will make things much easier. We click the configure button and click the wizard, then enter the details of the Jira instance that we are connecting to. We need to use a login email and an API token generated from the Jira account. 

First we go to profile and my settings. First we go to profile, then my settings, then the settings tab, where we will see the fields to enter our information and for details on how to do integration, check the description. Let's now check out how the integration works. We can mark a specific test as "failed" and add some additional details. Notice certain links in the defects field. We can also push across an issue tracker without having to leave a specific application. The fields present are pulled by the issue tracker directly. All of the projects, issue types, and components are defined in Jira, with dialects looking slightly different with each integration. We can also create a push template in order to push more testrail data into the issue description. 

To set this up, head to administration, then integration. Then we click a link for enter a template and see another text field appear. We can see the test settings on enter a template and we'll see a link for add field below the test field result to include a description of the issue pushing through the bug tracker. This includes any static text included in the template and even any in-line formatting options to create headings or change the appearance of the text. 

With push, if we push a new issue  to Jira, we see that all the fields will now display in the Jira issue description, and we can customize the fields through the issue tracker integration. Once we push things, test rail offers the new issue ids. You can also push multiple bug reports and manually add IDs if needed. 

We can also add a run directly from within a Jira issue, and we link test cases to external references, such as requirements buffer reports, use cases and feature requests. We can enter one or multiple IDs for existing references such as Jiras or issue requirements. We can add defects or references column as well for test cases. 

Now, we want to learn how to manage project milestones, test assignments and personal tasks, and learn about email notifications. Milestones help to track the targets or the important timeline of a testing goal, such as a public software release or testing phase, and using the tab with the same name. Milestones might be different depending on the type of target I want to track, and we can set a due date or mark a milestone as completed. 

Another thing to do is to assign specific test plans or test runs to the milestone, which will help view all tests from milestones at a glance. Milestones make it easy to track test runs and results in parallel. We can select the milestone and subsequently see the progress of this milestone to see the current overall status and the results of all of the linked test runs and provide an overview of test progress. Grouping the test runs sees a broad overview and provides an additional datapoint for testing metrics and test reports. 

To link a test run to a milestone, select the milestone from dropdown when adding or editing your run. Another important element are the Todos - the individual tasks for each member. If we go to a test run, we select a few tests and assign them to myself, and select those tasks from todo. 

We can see everyone's workload and assign new tasks accordingly and filter the todos and their current status, and we can directly change the status of each user page. 

In this way, we can save a lot of time, and users can subscribe to notifications in specific tests and entire test runs, and we can get to notifications on whether to subscribe or not. 

Next, we want to discuss about TestRail’s test plans. Test plans are collections of test runs that make it easier to create multiple runs at once, which is very useful for teams using multiple test suites. They can also be used to create multiple runs to test their software across many different configurations. We should recommend watching test runs before watching a specific video.

First, Do add test run, which gives the opportunity to create a name, to create a description, and to link things to a milestone. You can either add a test run, or a test suite, depending on if you are in a project using test runs or test suites, respectively.

We have the option to add the included test cases, assign these tasks to a user, and add a description to this run.

 If you click the option to select cases, you will be able to select individual tests in this run. You can also use configurations to create multiple runs simultaneously for testing in multiple environments. You can manually create test runs, or let test rail creat them automatically based on the parameters that you defined.

First, click configurations button and add a group, which is defined by a different environment and variable that I’ll be testing with things inside my plan.The first group will be called “browsers” and add configurations for the most commonly used browsers.

Under the test run, we all have identical test runs, and once we have the test plan, we go to the plan overview page and shows all of the metrics from each of the runs contained in the plan and allow me to combine multiple configuration groups. With the new configurations save, we update the testplan and we can uncheck and runs that may not be valid combinations.

Test plans are a great way to group test runs. It can be very helpful to group test plans with milestones to better manages testing efforts. We use milestones to separate relases and use test plans to further group test runs. We can define test plans with different icons to indicate that a particular result is for multiple test plans rather than one.

The next topic that we plan on discussing is the report and test metrics. We view real-time metrics and statistics and generate advanced testing reports with TestRail. TestRail comes with various reporting options and many of these options are built directly onto TestRail’s Interface. The metrics always show the latest status of the testing efforts in real time.

For example, when we open a testrun, the overview page shows the latest status report of the test run, and we can group test runs by status by clicking on the column header or adding new columns based on the different fields of a test. The activity page offers an insight of the activity of the test run over time, and highlighting all changes/test results that have been added during the life cycle of the test run.

Instead of showing the latest status of the test, the activity report shows all of the test results that will be added in the test run. We can also find the progress page that comes with a forecasting feature using the estimates and elapsed time data to estimate results when generating a forecast of the remaining effort co complete a run. We can get a good idea of how long the test will take to execute as a result.

The burndown chart of the progress page shows how much work is left to do. This way we have an idea on how much time that is needed to complete a test run. The burndown chart on the progress page shows how much work is left to do based on the time each test case takes to execute. We can find a projected completion date, along with various other test methods and statistics. The more tests, the higher accuracy these forecasts will be.

The final tab is the defects tab, the defects that will be linked when adding the test results. We can see the total tests, results, and defects logged or linked, and then we see a full list of issues that were manually added or created using the add or push functions. We can also hover over the issue ID link to see more details at a glance without viewing the issue directly through the defect tracker.

Defect reports are also available for both test plans and milestones. Using these pages we can track the status of linked issues and bugs, and overall progress of all the integrations in test rail. For additional reporting, we can head to the reports tab with built-in reporting templates making it easy to generate, schedule, and build statistics and metrics. There are templates to create summary reports or reports to see the coverage of bugs/requirements or comparison reports. To create milestone reports, use the summary reports.

There are plenty of options to customize the reporting output. The first step is the report options, where we can populate a very basic title, a description, and a milestone that I want to generate the report for. I can also select the metrics in the report that I would like to see included. We could set a time frame for the information that we want to be included in the report, so we can only display the information we choose.

The final tab is the tests tab, allowing to specify which test should be included based on a user-configured figure. Below this, I’ll actually see things for access and scheduling, and scheduline a report to generate at a certain instance, so I can create daily, weekly, or monthly reports automatically. We’re also allowed to trigger the creation of this report, useful for teams using automation scripts or tools. You can also enable notifications about the reports sent to testrail users or email to colleagues that don’t have testrail access in either PDF or HTML format, and testrail will either generate the report immediately or schedule it for the future.

Most reports should only take a few minutes to generate. We can see all the important metrics in this report regarding this milestone at a glance. These metrics include testing status, included testruns, the activity, and so on. The other report is the workload summary report, seeing the assigned tasks for each one of the testers.

If you want to see which test(s) have the highest failure rate, you can select status report with a list of test cases based on the number of test results received. You can use things to identify features that are the most problematic with the application.

Another report is the defect summary report, identifying any issue or defects in report, or we can directly import the reports from Jira to generate any details.

We always have the option to either print, download, or forward copie(s) of these reports. You can always try any other reporting option presented in the sidebar of reports tab. You can find a link to the documentation in order to create a custom report.  

 The next thing we need to go over is the TestRail’s Administration and Customization. Before we get started, the administration area in testrail is only used by users with specific administrative preferences.

The overview page have details about the background tests and Testrail versions, which is important with sending email reports. There is also a quick access sidebar in order to easily navigate to the next section.

Clicking on “Users and Roles” gives you the chance to edit your users, all your groups, and all your roles. We can determine which specific parts of testrail to interact with. There’s Read-only, tester, designer, and lead. This allows to add, edit, run test plans, and more. You can then view an update specific metrics about the user such as their name, email address, default time zone, and more.

We can update the user’s role to an access tab, mark them as active or inactive, and make them an administrator. When viewing a user’s profile on the right side of the sidebar there’s an option to force them to change the password and an option to forget a user.

You can also make user groups, allow to have specific groups for project teams or particular users such as clients. These are useful for granting permissions to users withint a group for specific projects. This brings to the next tab of the administration area.

When editing a project, go to the access tab to limit the permissions of specific group by updating the project permissions to a specific role. You can also configure project defects (issue/defect integration, etc) and you can overwrite these settings for specific projects. Now, let’s head to the integration tab.

You can also configure integrations with everything that TestRail supports. Another thing to do is to go to the site settings and configure global system settings, such as installation line,, default locals, and time zone.

In testrail, you can customize various aspects for your testing instances. You can use your own custom fields for the test cases and results, helping to track additional information for test. Approval status allows user to select single status for the case, and then selecting a field type after naming a field. Now, you can assign this custom field to one or multiple projects.

By default, custom fields are added to available projects and can add them to specific projects, with draft, approved or inactive. If we open a test case, we can see the new field is available to use. You can also use the customizations page to change the options available in various system fields, such as priority, test case type, and result status.

Users can add UI scripts to customize aspects of testrail, including the appearance or behavior using simplified CSS or JavaScript to add automation trigger for a TestRail script or tool.

In this video, we’ll learn all about TestRail’s enterprise edition and its options. Many of TestRail’s customers are some of the biggest companies in their field. Then, TestRail Enterprise is designed to meet the needs of larger organizations with enterprise-level features and support.

To begin with, TestRail enterprise begins with a single Sign-On feature or TestRail Enterprise, for support (SSO). This allow s the use of one set of login credentials to access multiple applications, to mitigate the management across multiple usernames and passwords. This allows administrators to integrate the software with the SSO identity provider through the SAML 2.0 Authentication protocol.  With this enterprise feature, administrators do not have to separate credentials for each application, they only have to provide them once.

We can setup SSO configuration in the site setting menu or the configuration. Then Enhanced API performance can have optimized architecture that supports 300 tests per minute, which means faster testruns and improved synchronization of testrun results.

Another feature of enterprise edition is the audit log. An audit functionality maintains a complete record of all changes made to entities within a testrail instance. Administrators can monitor every change on projects with cases whether they occur within testrail or externally. You can improve security by specifying logging levels, and setting retention policies.

The auditing configuration is available through the administration console, and can adjust the audit levels to control what changes are documented within the logs, as well as the number of rows within the audit, the maximum number of audit records stored, and the maximum number of days of the audit record.

Once auditing is on, you can monitor changes through the testrail instance by navigating through the Logs tab. This could be updates, deletions and creations. If you wish the export the log for more detailed analysis, you can do so through the export button. Another great feature is to select the own preferred backup time and trigger the own automatic restores. By default these periods are performed in early morning hours but we can trigger owned preferred windows to minimize any distraction. One thing is access to the priority support, and if you already have the standard edition, don’t hestitate to contact or reach out to testrail support.    

 The final point discusses the shared test steps in TestRail. The ability to share test steps is one of the most commonly requested new feature in TestRail. The main purpose of shared test steps is to diminish the duplication of work and make it easy to use the same set of test steps across multiple different test cases featuring the same set of steps. You can now make a single change in one centralized location and have that propagate to all steps.

A user needs to be assigned a role so that they could add or edit test cases.

There are 2 ways to create a set of shared steps. You can create a step from a repository, or you can share a steps you’ve already added to an existing test case. The shared test case button is at the top right of the test case list page. We click add shared test steps, and we need to name the set of steps, including the testing information. Once done, save the set of steps, which we can now import to test cases.    

Now let’s take a look on how to share a set of steps that are already defined on a test case. We open the test case that includes the steps and then go to the edit page. From here, click the shared button on one of the steps, and then click the create shared steps dialogue page. We can check the boxes for each of the steps we want to include, which is useful when we want to share selected steps in this case. Once the selection is made, add a name for the set of steps.

Now, let’s look at how to add shared steps to the test case. First, we want to make sure that we are using the correct test case step template and define all of the basic info for the test case. You can import and select the set of steps we want to get to the test case. If we want to add further steps outside of the shared steps, we can do that.

Let’s talk about editing our test cases with shared steps. If we want to edit the steps within a shared set, we go to the repository and edit the sets with the step, and we can see which test cases will be affected. We are unable to edit the step test cases included, any changes must be done directly within a shared steps repository.

Now you can customize the steps to include 2 new additional data points, including an entire extra text field with the label additional step information, which can be used for any information, including the expected result for each step. In addition, there’s a new option to add references to the individual test steps, tying to the defect tracker integrations.

If there are any specific steps associated with an external requirement, bug, or other entity, you can link an individual step to that entity in the external tool. These settings can be in the steps tool either globally or on a project-by-project basis. This covers all of the basics of the steps feature.