# The artefacts.yaml File

Site [llms.txt](/llms.txt)

---

In order to run tests, you will need to have a `artefacts.yaml` file setup in the root of your project. The configurations made in this file allows artefacts to:

* Connect to the corresponding organization and project on the artefacts dashboard.
* Provides details about a given job including:
    * The job name
    * How to build the project (if using `run --in-container` or `run-remote`)
    * What framework and simulator is required for the job to run
    * Metrics to collect
    * Parameters to use
    * What launch file to use

## Example Configuration

The below is an example `artefacts.yaml` configuration file taken from our [nav2 example repo](https://github.com/art-e-fact/navigation2_ignition_gazebo_example). Note that with this configuration, there are two jobs, named `basic` and `nav2` respectively.

Each section will be explained in further detail on this page.

```yaml
version: 0.1.0

project: artefacts/navigation2-ignition-example

jobs:

  basic: # Only checks that things are loading
    type: test
    package:
      docker:
        build:
          dockerfile: ./Dockerfile
    runtime:
      simulator: gazebo:fortress
      framework: ros2:humble
    timeout: 5 #minutes
    scenarios:
      defaults: # Global to all scenarios, and overriden in specific scenarios.
        output_dirs: ["output"]
      settings:
        - name: bringup
          pytest_file: "src/sam_bot_nav2_gz/test/test_bringup.py" # when using pytest or ros2 launch_pytest

  nav2:
    type: test
    package:
      docker:
        build:
          dockerfile: ./Dockerfile
    runtime:
      simulator: gazebo:fortress
      framework: ros2:humble
    timeout: 5 #minutes
    scenarios:
      defaults: # Global to all scenarios, and overriden in specific scenarios.
        output_dirs: ["output"]
        metrics:
          - /odometry_error
          - /distance_from_start_gt
          - /distance_from_start_est
        params:
          launch/world: ["bookstore.sdf", "empty.sdf"]
      settings:
        - name: reach_goal
          pytest_file: "src/sam_bot_nav2_gz/test/test_reach_goal.py" # when using pytest or ros2 launch_pytest
        - name: follow_waypoints
          launch_test_file: "src/sam_bot_nav2_gz/test/test_follow_waypoints.launch.py" # when using ros2 launch_test
```

To briefly summarize:

The first job `basic`:

* Will be built using the Dockerfile named `Dockerfile` at the root of the project respository
* Is running on ros2 humble, using gazebo (ignition) fortress as a simulator
* Will timeout if the test(s) do not come to completion after 5 minutes
* Will upload anything in the `output` directory to the Artefacts Dashboard after test completion
* Will run one test ("scenario") using the `test_bringup.launch.py` launch file.

The second job `nav2`:

* Will be built using the Dockerfile named `Dockerfile` at the root of the project respository
* Is running on ros2 humble, using gazebo (ignition) fortress as a simulator
* Will timeout if the test(s) do not come to completion after 5 minutes
* Will upload anything in the `output` directory to the Artefacts Dashboard after test completion
* Will display the three listed `metrics` in the Artefacts Dashboard.
* Has two parameters (two different world files), which in this case will be used as ros `launch_arguments`
* Will run a total of **4** tests in two scenarios: `reach_goal` will run twice using the `test_reach_goal` launch file (once for each world listed in `params`), and `follow_waypoints` twice using the `test_follow_waypoints` launch file (again once for each world)

## Configuration Breakdown

* `version` *Optional* The artefacts yaml format specification version.
* `project` The name of the associated project. Needs to be in the format `<organization>/<project>`
* `jobs` A mapping of job names to `Job` definitions, see [Job](#job)

### Jobs

```yaml
jobs:

  <job_name>:
    type: test
    package:
       ...
    runtime:
       ...
    timeout: 5 #minutes
    scenarios:
       ...
```
Each Job has the following properties:

* The name of the job 
* `type` Defaults  to `test`
* `package` *Optional* Use when configuring how to build the job if running in a container (`run --in-container` or `run-remote`). See [Package](#package)
* `runtime` Contains runtime properties, (the framework and simulator). See [Runtime](#runtime)
* `timeout` *Optional* Time before the job gets marked as `timed out`
* `scenarios` One job can contain multiple scenarios, usually a test suite linked to a particular environment, see [Scenario definition](#scenarios-definition)

### Package

> [!NB] _Used when running on Artefacts Cloud Simulation_

* `custom` Can be used to customize the default build flow of any given project. See [Packaging for Cloud Simulation](user-guide/cloud-simulation.md#packaging-for-cloud-simulation) for details

* `docker` Can be used to provide a dockerfile for artefacts to use when building the test environment. See [Packaging with Docker](user-guide/docker-package.md) for details

### Runtime

Used to prepare and hook into the test environment

* `framework` Software framework. Supported values: `ros2:humble`, `ros2:galactic`, `ros2:jazzy`, `null` (experimental)

* `simulator` Simulation engine. Supported values: `turtlesim`, `gazebo:fortress`, `gazebo:harmonic`

> [!NOTE]
> In many cases, the artefacts CLI will still be compatible with a framework / simulator not listed above when running locally. However, when running in artefacts cloud simulation, you must provide a `package` block, and either a set of [custom commands](user-guide/cloud-simulation.md#customizing-packaging-for-cloud-simulation), or a [dockerfile](user-guide/docker-package.md#specifying-a-dockerfile-or-docker-image).

### Scenarios definition

Referring to the example from the top of the page:
```yaml
scenarios:
  defaults: # Global to all scenarios, and overriden in specific scenarios.
    output_dirs: ["output"]
    metrics:
      - /odometry_error
      - /distance_from_start_gt
      - /distance_from_start_est
    params:
      launch/world: ["bookstore.sdf", "empty.sdf"]
  settings:
    - name: reach_goal
      pytest_file: "src/sam_bot_nav2_gz/test/test_reach_goal.py"
    - name: follow_waypoints
      launch_test_file: "src/sam_bot_nav2_gz/test/test_follow_waypoints.launch.py"
```
* `defaults` Contains default scenario settings common to all scenarios unless overwritten by a scenario in `settings`. In the example `output_dirs`, `metrics`, and the `params` configurations will be shared across both scenario `reach_goal` and `follow_waypoints`

* `settings` Contains a list of `scenario`s, with any configurations from `defaults` being overwritten. See [Scenario](#scenario) below for settings available.

#### Scenario

* `name` Name of the scenario

* One of `pytest_file` / `launch_test_file` / `run`

    * `pytest_file` (When using `pytest` or ROS2 `launch_pytest` testing framework): Path to your test file.
    * `launch_test_file` (When using ROS2 `launch_test` testing framework): Path to your test file (typically xxx.launch.py)
    * `run` Command string used to start tests (executed via `subprocess.run(command, shell=True)`). Typically for power users.

> [!NOTE]
> When using `run`, where possible please output your test results to a JUnit XML file named `tests_junit.xml` saved in `output_dirs`. Otherwise the dashboard will not correctly display final test result statuses (`pass` / `fail` / `error`)


* `output_dirs` *Optional* List of paths where the Artefacts client will look for artifacts to upload to the Dashboard.
Supported types include .html files (can be created with [plotly](https://plotly.com/python/interactive-html-export/), they will be rendered as interactive figures) and videos (we recommend h264/mp4 for cross-platform compatibility).

* `launch_arguments` *Optional* ROS only. Dictionary of arguments `name: value` pairs to pass to the launch file. Typically used to configure execution behavior, like whether to run headless or not, whether to record rosbags...

* `params` List of parameters to set for the scenario. For each parameter a list of values or a single value can be specified. Scenario variants will automatically be run for each of the parameters combination (grid strategy).  All test results will be uploaded in the same Dashboard entry.

    * For the ROS2 framework, parameter names must follow the convention `node_name/parameter_name` (delimited by a single forward slash), made available through the environment variable `ARTEFACTS_SCENARIO_PARAMS_FILE`, as well as being accessible to the artefacts toolkit. They can be used to control the behavior of nodes. Nested parameters are supported using the dot notation (e.g. `node_name/parameter_name.subparameter_name`).

    * (experimental) For the 'null' framework, parameter names will be set as environment variables (make sure that parameter names are only letters, numbers and underscores).

> [!NOTE]
> You can also list ros launch arguments in the `params` section replacing `<node_name>` for `launch` and accessing it via the artefacts toolkit's [`get_artefacts_param`](/artefacts-toolkit/config/#get_artefacts_param) helper function.


* `metrics` *Optional*  To specify test metrics. Accepts a json file: the key-values pairs will be used as metric_name/metric_value. ROS projects can alternatively accept a list of topics, the latest values on the topic(s) during a run will be the logged value.