Reports¶

Reports are configurable summaries after a Robocop scan. For example, it can display a total number of issues discovered. They are dynamically loaded during setup according to a configuration.

Report class collects diagnostic messages from a linter and can optionally parse them. At the end of the scan it will generate the report.

To enable report use -r / --reports argument and provide the name of the report. You can use multiple reports with separate arguments (-r report1 -r report2) or comma-separated list (-r report1,report2). For example:

Cli

robocop check --reports rules_by_id,some_other_report

Configuration file

[tool.robocop.lint]
reports = [
    "rules_by_id",
    "some_other_report"
]

Note

Reports can be only enabled and configured from the configuration file closest to the current working directory. If you configure reports in multiple configuration files, only one configuration file will apply.

To enable all default reports use --reports all. Non-default reports can be only enabled using their name.

The order of the reports is preserved. For example, if you want timestamp report to be printed before any other reports, you can use the following configuration:

Cli

robocop check --reports timestamp,all

Configuration file

[tool.robocop.lint]
reports = [
    "timestamp",
    "all"
]

List available reports¶

Print a list of all reports with their configured status by using list reports command:

robocop list reports --reports all

You can filter the list using --enabled / --disabled flags:

robocop list reports --reports timestamp,sarif --disabled

Configuring reports¶

It is possible to configure some of the reports using --configure (or -c) option followed by report name, its parameter and the value:

robocop check --configure <report_name>.<param_name>=<value>

For example:

Cli

robocop check --configure sarif.report_filename=robocop_sarif.json --reports sarif

Configuration file

[tool.robocop.lint]
reports = [
    "sarif"
]
configure = [
    "sarif.report_filename=robocop_sarif.json"
]

configures report_filename parameter to robocop_sarif.json for sarif report.

Disable all reports¶

When handling multiple configuration sources it may be possible to inherit reports configuration that we don’t want to use. Use special keyword None to not run any reports even if configured:

robocop check --reports sarif,all,None

Comparing results¶

Several reports allows to compare current run with the previous run. Use compare_runs report name to enable it. # TODO: make it an option

Example output:

Found 18 (-3) issues: 13 (-4) INFOs, 5 (+1) WARNINGs.

Issues by ID:
MISC12 [I] (unnecessary-string-conversion)    : 10 (+0)
NAME01 [W] (not-allowed-char-in-name)         : 2 (+0)
VAR02 [I] (unused-variable)                   : 2 (-4)
VAR03 [W] (variable-overwritten-before-usage) : 2 (+1)
TAG05 [I] (could-be-test-tags)                : 1 (+0)
VAR11 [W] (overwriting-reserved-variable)     : 1 (+0)

Robocop stores previous result in cache directory. Cache directory is stored in the different location depending on the platform:

Linux: "~/.cache/robocop"
macOS: "~/Library/Caches/robocop"
Windows: "C:\\Users\\<username>\\AppData\\Local\\robocop"

Saving the results is disabled by default and can be enabled with --persistent flag:

Cli

robocop check --persistent

Configuration file

[tool.robocop.lint]
persistent = true

Only the previous run for the current working directory is saved.

To used stored results to compare with current run, enable compare_runs report:

Cli

robocop check --reports all,compare_runs

Configuration file

[tool.robocop.lint]
reports = [
    "all",
    "compare_runs"
]

Reports list¶

Print issues¶

Report name: print_issues

This report is always enabled. Report that collect diagnostic messages and print them at the end of the execution.

There are available three different types of output:

extended (default), which print issue together with source code:

test.robot:2:14 DEPR02 'Run Keyword If' is deprecated since Robot Framework version 5.*, use 'IF' instead
   |
 1 | *** Settings ***
 2 | Suite Setup  Run Keyword If
   |              ^^^^^^^^^^^^^^ DEPR02
 3 | Suite Teardown  Run Keyword If
 4 | Force Tags         tag
   |

grouped, which prints issues grouped by source files:

templated_suite.robot:
  1:1 MISC06 No tests in 'templated_suite.robot' file, consider renaming to 'templated_suite.resource' (can-be-resource-file)
  2:18 DEPR02 'Run Keyword Unless' is deprecated since Robot Framework version 5.*, use 'IF' instead (deprecated-statement)

test.robot:
  1:1 DOC03 Missing documentation in suite (missing-doc-suite)
  3:17 DEPR02 'Run Keyword If' is deprecated since Robot Framework version 5.*, use 'IF' instead (deprecated-statement)

simple, which print issue in one line. It also allows to configure format of message:

variable_errors.robot:7:1 [E] ERR01 Robot Framework syntax error: Invalid dictionary variable item '1'. Items must use 'name=value' syntax or be dictionary variables themselves.
positional_args.robot:3:32 [E] ERR01 Robot Framework syntax error: Positional argument '${arg2}' follows named argument

You can configure output type with output_format:

robocop check --configure print_issues.output_format=grouped

Format of simple output type can be configured with --issue-format option:

robocop check --issue-format "{source}:{line}:{col} [{severity}] {rule_id} {desc} ({name})"

Format of extended output type can be configured with issue_format parameter:

robocop check --configure print_issues.issue_format="{source}"

Rules by ID¶

Report name: rules_by_id

Report that groups linter rules messages by rule id and prints it ordered by most common message. Example:

Issues by ID:
VAR03 [W] (variable-overwritten-before-usage) : 5
DOC01 [W] (missing-doc-keyword)               : 4
ERR01 [E] (parsing-error)                     : 3
NAME01 [W] (not-allowed-char-in-name)         : 2
MISC01 [W] (keyword-after-return)             : 1

Rules by severity¶

Report name: rules_by_error_type

Report that groups linter rules messages by severity and prints total of issues per every severity level.

Example:

Found 15 issues: 4 ERRORs, 11 WARNINGs.

Return status¶

Report name: return_status

Report that checks if number of returned rules messages for given severity value does not exceed preset threshold. If the report is enabled, it will be used as a return status from Robocop.

Execution time¶

Report name: scan_timer

Report that returns Robocop execution time

Example:

Scan finished in 0.054s.

File statistics¶

Report name: file_stats

Report that displays overall statistics about number of processed files.

Example:

Processed 7 files from which 5 files contained issues.

Robocop version¶

Report name: version

Report that returns Robocop version.

Example:

Report generated by Robocop version: 2.0.2

Report timestamp¶

Report name: timestamp

Report that returns Robocop execution timestamp. Timestamp follows local time in format of Year-Month-Day Hours(24-hour clock):Minutes:Seconds ±hh:mm UTC offset as default.

Example:

Reported: 2022-07-10 21:25:00 +0300

Both of default values, timezone and format can be configured by -c/--configure and timestamp:timezone:"<timezone name>" and/or timestamp:format:"<format string>":

robocop check -c timestamp.timezone="Europe/Paris" -c timestamp.format="%Y-%m-%d %H:%M:%S %Z %z"

This yields following timestamp report:

Reported: 2022-07-10 20:38:10 CEST +0200

For timezone names, see here.

For timestamp formats, see datetime format codes.

Useful configurations:

Local time to ISO 8601 format:
robocop check --configure timestamp.format="%Y-%m-%dT%H:%M:%S%z"

UTC time:
robocop check --configure timestamp:timezone:"UTC" --configure timestamp.format="%Y-%m-%dT%H:%M:%S %Z %z"

Timestamp with high precision:
robocop check --configure timestamp.format="%Y-%m-%dT%H:%M:%S.%f %z"

12-hour clock:
robocop check --configure timestamp.format="%Y-%m-%d %I:%M:%S %p %Z %z"

More human-readable format 'On 10 July 2022 07:26:24 +0300':
robocop check --configure timestamp.format="On %d %B %Y %H:%M:%S %z"

JSON export¶

Report name: json_report

Report that returns a list of found issues in a JSON format. The output file will be generated in the current working directory with the robocop.json name.

This report is not included in the default reports. The --reports all option will not enable this report. You can still enable it using report name directly: --reports json_report or --reports all,json_report.

You can configure output path. It’s relative path to file that will be produced by the report:

robocop check --configure json_report.output_path=C:/json_reports/report.json

Default path is robocop.json .

Example content of the file:

[
    {
        "source": "C:\robot_tests\keywords.robot",
        "line": 1,
        "end_line": 1,
        "column": 1,
        "end_column": 1,
        "severity": "I",
        "rule_id": "0913",
        "description": "No tests in 'keywords.robot' file, consider renaming to 'keywords.resource'",
        "rule_name": "can-be-resource-file"
    },
    {
        "source": "C:\\robot_tests\\keywords.robot",
        "line": 51,
        "end_line": 51,
        "column": 1,
        "end_column": 13,
        "severity": "W",
        "rule_id": "0324",
        "description": "Variable '${TEST_NAME}' overwrites reserved variable '${TEST_NAME}'",
        "rule_name": "overwriting-reserved-variable"
    }
]

SARIF export¶

Report name: sarif

Report that generates SARIF output file.

This report is not included in the default reports. The --reports all option will not enable this report. You can still enable it using report name directly: --reports sarif or --reports all,sarif.

All fields required by GitHub Code Scanning are supported. The output file will be generated in the current working directory with the .sarif.json name.

You can configure output path. It’s relative path to file that will be produced by the report:

robocop check --configure sarif.output_path=output/robocop_sarif.json

Default path is .sarif.json .

Gitlab¶

Report name: gitlab

Report that generates Gitlab Code Quality output file.

This report is not included in the default reports. The --reports all option will not enable this report. You can still enable it using report name directly: --reports gitlab or --reports all,gitlab.

It allows to display issue information in the Gitlab, for example in the PR view. More information at https://docs.gitlab.com/ci/testing/code_quality/#code-quality-report-format .

You can configure output path. It’s relative path to file that will be produced by the report:

robocop check --configure gitlab.output_path=output/robocop_code_quality.json

Default path is robocop-code-quality.json .

Sonar Qube¶

Report name: sonarqube

Report that generates SonarQube report.

This report is not included in the default reports. The --reports all option will not enable this report. You can still enable it using report name directly: --reports sonarqube or --reports all,sonarqube.

Implements all mandatory field of Sonar Qube Generic formatted issue report.

Currently, Robocop supports 2 major minimal Sonar Qube versions:

up to 9.9
10.3 onwards (default)

If your SonarQube instance requires 9.9 format (which for example doesn’t have impacts field), you can configure it using sonar_version parameter:

robocop check --configure sonarqube.sonar_version=9.9

You can configure output path. It’s relative path to file that will be produced by the report:

robocop check --configure sonarqube.output_path=output/robocop_sonar_qube.json

Default path is robocop_sonar_qube.json .