Reports¶
Reports are configurable summaries after a Robocop scan. For example, it can display a total number of issues discovered. They are dynamically loaded during setup according to a configuration.
Report class collects diagnostic messages from a linter and can optionally parse them. At the end of the scan it will generate the report.
To enable report use -r
/ --reports
argument and provide the name of the report.
You can use multiple reports with separate arguments (-r report1 -r report2
) or comma-separated list
(-r report1,report2
). For example:
robocop check --reports rules_by_id,some_other_report
[tool.robocop.lint]
reports = [
"rules_by_id",
"some_other_report"
]
Note
Reports can be only enabled and configured from the configuration file closest to the current working directory. If you configure reports in multiple configuration files, only one configuration file will apply.
To enable all default reports use --reports all
. Non-default reports can be only enabled using their name.
The order of the reports is preserved. For example, if you want timestamp
report to be printed before any
other reports, you can use the following configuration:
robocop check --reports timestamp,all
[tool.robocop.lint]
reports = [
"timestamp",
"all"
]
List available reports¶
Print a list of all reports with their configured status by using list reports
command:
robocop list reports --reports all
You can filter the list using --enabled
/ --disabled
flags:
robocop list reports --reports timestamp,sarif --disabled
Configuring reports¶
It is possible to configure some of the reports using --configure
(or -c
) option followed by report name,
its parameter and the value:
robocop check --configure <report_name>.<param_name>=<value>
For example:
robocop check --configure sarif.report_filename=robocop_sarif.json --reports sarif
[tool.robocop.lint]
reports = [
"sarif"
]
configure = [
"sarif.report_filename=robocop_sarif.json"
]
configures report_filename
parameter to robocop_sarif.json
for sarif
report.
Disable all reports¶
When handling multiple configuration sources it may be possible to inherit reports configuration that we don’t want to
use. Use special keyword None
to not run any reports even if configured:
robocop check --reports sarif,all,None
Comparing results¶
Several reports allows to compare current run with the previous run. Use compare_runs
report name to enable it. # TODO: make it an option
Example output:
Found 18 (-3) issues: 13 (-4) INFOs, 5 (+1) WARNINGs.
Issues by ID:
MISC12 [I] (unnecessary-string-conversion) : 10 (+0)
NAME01 [W] (not-allowed-char-in-name) : 2 (+0)
VAR02 [I] (unused-variable) : 2 (-4)
VAR03 [W] (variable-overwritten-before-usage) : 2 (+1)
TAG05 [I] (could-be-test-tags) : 1 (+0)
VAR11 [W] (overwriting-reserved-variable) : 1 (+0)
Robocop stores previous result in cache directory. Cache directory is stored in the different location depending on the platform:
Linux:
"~/.cache/robocop"
macOS:
"~/Library/Caches/robocop"
Windows:
"C:\\Users\\<username>\\AppData\\Local\\robocop"
Saving the results is disabled by default and can be enabled with --persistent
flag:
robocop check --persistent
[tool.robocop.lint]
persistent = true
Only the previous run for the current working directory is saved.
To used stored results to compare with current run, enable compare_runs
report:
robocop check --reports all,compare_runs
[tool.robocop.lint]
reports = [
"all",
"compare_runs"
]
Reports list¶
Print issues¶
Report name: print_issues
This report is always enabled. Report that collect diagnostic messages and print them at the end of the execution.
There are available three different types of output:
extended (default), which print issue together with source code:
test.robot:2:14 DEPR02 'Run Keyword If' is deprecated since Robot Framework version 5.*, use 'IF' instead | 1 | *** Settings *** 2 | Suite Setup Run Keyword If | ^^^^^^^^^^^^^^ DEPR02 3 | Suite Teardown Run Keyword If 4 | Force Tags tag |
grouped, which prints issues grouped by source files:
templated_suite.robot: 1:1 MISC06 No tests in 'templated_suite.robot' file, consider renaming to 'templated_suite.resource' (can-be-resource-file) 2:18 DEPR02 'Run Keyword Unless' is deprecated since Robot Framework version 5.*, use 'IF' instead (deprecated-statement) test.robot: 1:1 DOC03 Missing documentation in suite (missing-doc-suite) 3:17 DEPR02 'Run Keyword If' is deprecated since Robot Framework version 5.*, use 'IF' instead (deprecated-statement)
simple, which print issue in one line. It also allows to configure format of message:
variable_errors.robot:7:1 [E] ERR01 Robot Framework syntax error: Invalid dictionary variable item '1'. Items must use 'name=value' syntax or be dictionary variables themselves. positional_args.robot:3:32 [E] ERR01 Robot Framework syntax error: Positional argument '${arg2}' follows named argument
You can configure output type with output_format
:
robocop check --configure print_issues.output_format=grouped
Format of simple output type can be configured with --issue-format
option:
robocop check --issue-format "{source}:{line}:{col} [{severity}] {rule_id} {desc} ({name})"
Format of extended output type can be configured with issue_format
parameter:
robocop check --configure print_issues.issue_format="{source}"
Rules by ID¶
Report name: rules_by_id
Report that groups linter rules messages by rule id and prints it ordered by most common message. Example:
Issues by ID:
VAR03 [W] (variable-overwritten-before-usage) : 5
DOC01 [W] (missing-doc-keyword) : 4
ERR01 [E] (parsing-error) : 3
NAME01 [W] (not-allowed-char-in-name) : 2
MISC01 [W] (keyword-after-return) : 1
Rules by severity¶
Report name: rules_by_error_type
Report that groups linter rules messages by severity and prints total of issues per every severity level.
Example:
Found 15 issues: 4 ERRORs, 11 WARNINGs.
Return status¶
Report name: return_status
Report that checks if number of returned rules messages for given severity value does not exceed preset threshold. If the report is enabled, it will be used as a return status from Robocop.
Execution time¶
Report name: scan_timer
Report that returns Robocop execution time
Example:
Scan finished in 0.054s.
File statistics¶
Report name: file_stats
Report that displays overall statistics about number of processed files.
Example:
Processed 7 files from which 5 files contained issues.
Robocop version¶
Report name: version
Report that returns Robocop version.
Example:
Report generated by Robocop version: 2.0.2
Report timestamp¶
Report name: timestamp
Report that returns Robocop execution timestamp.
Timestamp follows local time in format of
Year-Month-Day Hours(24-hour clock):Minutes:Seconds ±hh:mm UTC offset
as default.
Example:
Reported: 2022-07-10 21:25:00 +0300
Both of default values, timezone
and format
can be configured by
-c/--configure
and timestamp:timezone:"<timezone name>"
and/or timestamp:format:"<format string>"
:
robocop check -c timestamp.timezone="Europe/Paris" -c timestamp.format="%Y-%m-%d %H:%M:%S %Z %z"
This yields following timestamp report:
Reported: 2022-07-10 20:38:10 CEST +0200
For timezone names, see here.
For timestamp formats, see datetime format codes.
Useful configurations:
Local time to ISO 8601 format:
robocop check --configure timestamp.format="%Y-%m-%dT%H:%M:%S%z"
UTC time:
robocop check --configure timestamp:timezone:"UTC" --configure timestamp.format="%Y-%m-%dT%H:%M:%S %Z %z"
Timestamp with high precision:
robocop check --configure timestamp.format="%Y-%m-%dT%H:%M:%S.%f %z"
12-hour clock:
robocop check --configure timestamp.format="%Y-%m-%d %I:%M:%S %p %Z %z"
More human-readable format 'On 10 July 2022 07:26:24 +0300':
robocop check --configure timestamp.format="On %d %B %Y %H:%M:%S %z"
JSON export¶
Report name: json_report
Report that returns a list of found issues in a JSON format. The output file will be generated
in the current working directory with the robocop.json
name.
This report is not included in the default reports. The --reports all
option will not enable this report.
You can still enable it using report name directly: --reports json_report
or --reports all,json_report
.
You can configure output path. It’s relative path to file that will be produced by the report:
robocop check --configure json_report.output_path=C:/json_reports/report.json
Default path is robocop.json
.
Example content of the file:
[
{
"source": "C:\robot_tests\keywords.robot",
"line": 1,
"end_line": 1,
"column": 1,
"end_column": 1,
"severity": "I",
"rule_id": "0913",
"description": "No tests in 'keywords.robot' file, consider renaming to 'keywords.resource'",
"rule_name": "can-be-resource-file"
},
{
"source": "C:\\robot_tests\\keywords.robot",
"line": 51,
"end_line": 51,
"column": 1,
"end_column": 13,
"severity": "W",
"rule_id": "0324",
"description": "Variable '${TEST_NAME}' overwrites reserved variable '${TEST_NAME}'",
"rule_name": "overwriting-reserved-variable"
}
]
SARIF export¶
Report name: sarif
Report that generates SARIF output file.
This report is not included in the default reports. The --reports all
option will not enable this report.
You can still enable it using report name directly: --reports sarif
or --reports all,sarif
.
All fields required by GitHub Code Scanning are supported. The output file will be generated
in the current working directory with the .sarif.json
name.
You can configure output path. It’s relative path to file that will be produced by the report:
robocop check --configure sarif.output_path=output/robocop_sarif.json
Default path is .sarif.json
.
Gitlab¶
Report name: gitlab
Report that generates Gitlab Code Quality output file.
This report is not included in the default reports. The --reports all
option will not enable this report.
You can still enable it using report name directly: --reports gitlab
or --reports all,gitlab
.
It allows to display issue information in the Gitlab, for example in the PR view. More information at https://docs.gitlab.com/ci/testing/code_quality/#code-quality-report-format .
You can configure output path. It’s relative path to file that will be produced by the report:
robocop check --configure gitlab.output_path=output/robocop_code_quality.json
Default path is robocop-code-quality.json
.
Sonar Qube¶
Report name: sonarqube
Report that generates SonarQube report.
This report is not included in the default reports. The --reports all
option will not enable this report.
You can still enable it using report name directly: --reports sonarqube
or --reports all,sonarqube
.
Implements all mandatory field of Sonar Qube Generic formatted issue report.
Currently, Robocop supports 2 major minimal Sonar Qube versions:
up to 9.9
10.3 onwards (default)
If your SonarQube instance requires 9.9 format (which for example doesn’t have impacts field), you can configure it
using sonar_version
parameter:
robocop check --configure sonarqube.sonar_version=9.9
You can configure output path. It’s relative path to file that will be produced by the report:
robocop check --configure sonarqube.output_path=output/robocop_sonar_qube.json
Default path is robocop_sonar_qube.json
.