feat(tools): add idf.py diag reporting tool

The initial implementation of a diagnostic tool that collects valuable
information about esp-idf and failed build to assist in investigating
reported issues.

The gathered information includes environmental variables, details about
the python virtual environment, installed tools, platform information,
project_description.json, sdkconfig, build logs, map file, linker
scripts, and others.

usage:

1) create the default report

   # allow diag to create the report directory name
   $ idf.py diag
   # explicitly specify the report directory
   $ idf.py diag --output <report directory>

2) examine the contents of the generated <report directory> for
   sensitive information and add additional content to the
   <report directory>

3) create report archive zip file that can be shared or attached to
   the reported issue

   $ idf.py diag --zip <report directory>

The tool collects information as described in what are known as recipe
files. A recipe file is a YAML file, similar to an Ansible playbook or a
GitHub action, but much more simplified. Each recipe outlines how to
gather a set of related information. For instance, the manager.yml
recipe gathers data related to the component manager. Each recipe
includes metadata such as its description, tags, and steps. Tags are
used to determine which recipes to use; by default, all built-in recipes
located in tools/idf_py_actions/diag/recipes are used. Steps consist of
a list of commands to be executed. Currently, there are four commands:
file, exec, env, and glob. For more detailed information about recipes,
their format, and commands, please refer to
tools/idf_py_actions/diag/recipes/README.md.

Recipe example for component manager:

description: IDF Component Manager information
tags: [manager, base, project]
output: manager
steps:
  - name: 'IDF Component Manager'
    cmds:
      - exec:
        cmd: 'python -m idf_component_manager version'
        output: manager.ver
      - file:
        path: '${PROJECT_DIR}/dependencies.lock'
      - glob:
        # Gather all idf_component.yml files from the project directory and
        # save them in directories relative to the project directory within
        # the idf_component directory.
        pattern: 'idf_component.yml'
        recursive: true
        relative: true
        path: '${PROJECT_DIR}'
        output: 'idf_component/'

Create report for manager

1) all recipes with manager tag

   $ idf.py diag --tag manager

2) use only the manager recipe explicitly; built-in recipes can be
   referenced simply by their name, but all recipes can be referenced
   by their path

   $ idf.py diag --recipe manager
   or
   $ idf.py diag --recipe <full path>

To display available recipes, use

   $ idf.py diag --list

and to verify recipes, use

   $ idf.py diag --check

Both --list and --check honers the --tag and --recipe options.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>

.gitlab/ci/host-test.yml

@@ -209,6 +209,8 @@ test_tools:
     - pytest --noconftest test_mkdfu.py        --junitxml=${IDF_PATH}/XUNIT_MKDFU.xml                     || stat=1
     - cd ${IDF_PATH}/tools/test_idf_size
     - pytest --noconftest test_idf_size.py     --junitxml=${IDF_PATH}/XUNIT_IDF_SIZE.xml                  || stat=1
+    - cd ${IDF_PATH}/tools/test_idf_diag
+    - pytest --noconftest test_idf_diag.py     --junitxml=${IDF_PATH}/XUNIT_IDF_DIAG.xml                  || stat=1
     - cd ${IDF_PATH}
     - shellcheck -s sh tools/detect_python.sh                                                             || stat=1
     - shellcheck -s bash tools/detect_python.sh                                                           || stat=1

.gitlab/ci/rules.yml

@@ -92,6 +92,8 @@
   - "tools/idf_size.py"
   - "tools/test_idf_size/**/*"
 
+  - "tools/test_idf_diag/**/*"
+
   - "tools/tools.json"
   - "tools/tools_schema.json"
   - "tools/idf_tools.py"

tools/idf_py_actions/diag/purge.yml

@@ -0,0 +1,3 @@
+# Remove user, password, token from URL
+- regex: '://[^@]+@'
+  repl: '://[REDACTED]@'

tools/idf_py_actions/diag/recipes/README.md

@@ -0,0 +1,288 @@
+# Recipe format description for idf.py diag
+
+The `idf.py diag` command processes one or more `recipe` files. Each `recipe`
+file outlines a collection of related data and files that should be gathered.
+For instance, the `idf.yml` recipe gathers information related to the ESP-IDF.
+`Recipes` are formatted in YAML. The output from each `recipe` consists of a
+set of files located in the diagnostic report directory, which may be copied or
+generated as a result of executing certain commands.
+
+A `recipe` is made up of one or more `steps`, and each `step` contains one or
+more commands. A recipe can consist of just one `step` where all the commands
+are carried out. The aim is to split the `recipe` into logical steps if it is
+convenient. For instance, a `project.yml` `recipe` might include one `step` to
+gather all log files and another to collect linker script files, although these
+tasks could also be completed in a single `step`. Refer to the `recipes` in
+this directory for examples.
+
+## Overview of Recipe Structure
+
+    description: brief recipe description
+    tags:        list of tags that can be used to identify the recipe
+    output:      root recipe directory for its output in the report folder
+    steps:       list of steps to execute within a recipe
+      - name:    brief step description
+        output:  step directory for its output in the report folder
+        cmds:    list of commands to execute within a step
+          - cmd: command name
+            arg: command argument
+            ...
+          ...
+      ...
+
+## Recipe variables
+
+The `recipe` can utilize the following variables. The `idf.py diag` assigns
+values to these variables and expands them in the recipe upon loading. To use a
+variable, format it as `${NAME}`, such as `${IDF_PATH}`.
+
+* PROJECT_DIR
+
+    Project directory specified by idf.py using the `-C` or `--project-dir`
+    option.
+
+* BUILD_DIR
+
+    Build directory specified by idf.py using the `-B` or `--build-dir` option.
+
+* IDF_PATH
+
+    IDF path as defined in the environment variable.
+
+* REPORT_DIR
+
+    The report directory is where all the recipe outputs are stored. Keep in
+    mind that during the execution of `idf.py diag`, it points to a temporary
+    directory, which is moved to its final destination once the report is
+    successfully completed.
+
+## Recipe
+
+* description: string (required)
+
+    Short `recipe` description, which is shown in `idf.py diag` progress.
+
+* tags: list (optional)
+
+    Strings which identify this `recipe`. Used to specify which `recipes`
+    should the `idf.py diag` use. All `recipes` with a given tag are used.
+
+* output: string (optional)
+
+    Global output directory for the `recipe`. This directory serves as the main
+    directory for all files produced by this `recipe` within the report
+    directory.  For instance, if it is set to `idf` and the report directory is
+    `report`, then all files collected by this `recipe` will be stored in the
+    `report/idf` directory. This helps organize the collected files within the
+    report directory.
+
+* steps: list (required)
+
+    One or more `steps` to follow for this `recipe`. This allows, but does not
+    require, splitting the `recipe` into more logical sections. For example,
+    one `step` could gather all log files, while another might collect
+    environment information.
+
+## Step
+
+* name: string (required)
+
+    Brief description of the `step`, displayed in the `idf.py diag` progress
+    beneath the `recipe` description.
+
+* output: string (optional)
+
+    Global output directory for the `step`. This directory serves as the main
+    directory for all files produced by this `step` within the recipe `output`
+    directory.  For instance, if it is set to `logs` and the report directory
+    is `report` and recipe `output` is `idf`, then all files collected by this
+    `step` will be stored in the `report/idf/logs` directory. 
+
+* cmds: list (required)
+
+    Sequence of commands to be executed in this `step`.
+
+## Commands
+
+The following commands can be used within a `step` in the `recipe`.  Each
+command consists of a list that includes the command name, such as `exec` or
+`glob`, along with its arguments. Please be aware that if a command fails, it
+does not terminate `idf.py diag`; all commands will still be executed.  The
+command mapping key has no value, and if it is present, it is ignored.
+
+### file
+
+Copy the specified file from the given path to the report directory. If no
+`output` argument is provided, the file is copied using its original name.
+
+* path: string (required)
+
+    Path to the source file.
+
+* output: string (optional)
+
+    The destination path for the file within the report directory. If it ends
+    with a `/` character, it is treated as a directory, and the source file
+    name from the `path` argument is added to it. Otherwise, it is considered a
+    complete path including the file name. If not provided, the source file
+    name is used as the destination file name. The complete destination path in
+    the report directory is built starting with the `output` directory
+    specified in the `recipe`, followed by the `output` directory specified in
+    the`step`, and finally the `output` specified in the `file` command, if
+    each is provided. All directories that do not exist in the output path are
+    created.
+
+Example:
+
+    - file:
+      path: '${BUILD_DIR}/compile_commands.json'
+      output: 'build/json_files/commands.json'
+
+### exec
+
+Run the given command and save its output.
+
+* cmd: string or list (required)
+
+    Command to run. If it's a string, it starts via the shell; if it's a list,
+    it starts without shell expansions. Using the list format can help avoid
+    escaping command arguments.
+
+* output: string (optional)
+
+    The path in the report directory where the command's standard output should
+    be stored. If not specified, the command output will not be saved. The
+    complete destination path in the report directory is built starting with
+    the `output` directory specified in the `recipe`, followed by the `output`
+    directory specified in the `step`, and finally the `output` specified in
+    the `exec` command, if each is provided.  The `/` character at the end is
+    disregarded in contrast to the `file` command.
+
+* stderr: string (optional)
+
+    Similar to `output`, but specifically for standard error output.
+
+* timeout: int (optional)
+
+    The number of seconds to wait for the command to execute.
+
+* append: boolean (optional)
+
+    Append the command's standard output and standard error to the files
+    specified by the `output` and `stderr` arguments.
+
+Example:
+
+    - exec:
+      cmd: 'idf.py --version'
+      timeout: 10
+      output: esp_idf.ver
+
+    - exec:
+      cmd:
+        - python
+        - -c
+        - |
+          import platform
+          print(f'system: {platform.system()}')
+
+### env
+
+Store details of the specified environment variables in a file using the format
+variable=value.
+
+* vars: list (required)
+
+    A list of names for environment variables to gather.
+
+* regex: string (optional)
+
+    Optional regular expression to gather environment variables that match it.
+
+* output: string (optional)
+
+    The path in the report directory where the environment variable information
+    should be stored. If not specified, nothing will be saved. The complete
+    destination path in the report directory is built starting with the
+    `output` directory specified in the `recipe`, followed by the `output`
+    directory specified in the `step`, and finally the `output` specified in
+    the `env` command, if each is provided. The `/` character at the end is
+    disregarded, in contrast to the `file` command.
+
+* append: boolean (optional)
+
+    Append the environmental variables information to the file specified by the
+    `output` argument.
+
+Example:
+
+    - env:
+      vars:
+        - IDF_PATH
+        - IDF_PYTHON_ENV_PATH
+        - IDF_TOOLS_PATH
+        - PATH
+        - OPENOCD_SCRIPTS
+        - PYTHONPATH
+        - MSYSTEM
+      regex: '.*IDF.*|.*ESP.*'
+      output: environment.var
+
+### glob
+
+Find pathnames that match a specific pattern within a specified directory, and
+include them in the report directory.
+
+* pattern: string (required)
+
+    Pattern matching with wildcards.
+
+* path: string (required)
+
+    Directory to look for files that match the `pattern`.
+
+* output: string (optional)
+
+    Every file that matches the `pattern` is stored in the report directory at
+    the location specified by `output`. If `output` ends with a `/`, it is
+    interpreted as a directory, and the matched file name is appended to it. If
+    it does not end with a `/`, it is regarded as a full path including the
+    file name. If `output` is not specified, the name of the matching file is
+    used as the destination file name.
+
+    The full destination path in the report directory for each matched file
+    is constructed by using the `output` directory from the `recipe`, followed
+    by the `output` directory from the `step`, and finally the `output` from
+    the `glob` command, if each is available. Any directories that do not exist
+    in the output path are created. If the `relative` argument is set and the
+    `output` argument in the `glob` command ends with `/`, the file's relative
+    path to the `path` argument is appended to the `output` argument in the
+    `glob` command. A `.<cnt>` suffix is added to a file path if that path
+    already exists in the report directory. The `<cnt>` is a number that
+    increases until a unique path is found in the report directory.
+
+* mtime: boolean (optional)
+
+    If multiple files are found, add only the one that was modified most recently.
+
+* recursive: boolean (optional)
+
+    Recursively search the `path`.
+
+* relative: boolean (optional)
+
+    Save the file in the `output` directory, preserving the same relative path
+    it has in the `path` directory.
+
+* regex: string (optional)
+
+    Only add files whose content matches the specified regular expression.
+
+Example:
+
+    - glob:
+      pattern: 'idf_py_stdout_output*'
+      path: '${BUILD_DIR}/log'
+      regex: '^Command: .*idf_monitor.py.*$'
+      output: 'logs/monitor/'
+      mtime: True

tools/idf_py_actions/diag/recipes/environment.yml

@@ -0,0 +1,23 @@
+description: Shell environment information
+tags: [environment, base, project]
+output: environment
+steps:
+  - name: 'Enviromental Variables'
+    cmds:
+      - env:
+        vars:
+          - IDF_PATH
+          - IDF_PYTHON_ENV_PATH
+          - IDF_TOOLS_PATH
+          - PATH
+          - OPENOCD_SCRIPTS
+          - PYTHONPATH
+          - MSYSTEM
+        regex: '.*IDF.*|.*ESP.*'
+        output: environment.var
+
+  - name: 'Python Environmental Variables'
+    cmds:
+      - env:
+        regex: '.*PYTHON.*'
+        output: python.var

tools/idf_py_actions/diag/recipes/idf.yml

@@ -0,0 +1,29 @@
+description: Basic ESP-IDF information
+tags: [idf, base, project]
+output: idf
+steps:
+  - name: 'ESP-IDF Version'
+    cmds:
+      - exec:
+        cmd: 'idf.py --version'
+        timeout: 10
+        output: esp_idf.ver
+
+  - name: 'ESP-IDF Git Version'
+    cmds:
+      - exec:
+        cmd: 'git -C ${IDF_PATH} describe'
+        output: esp_idf_git.ver
+
+  - name: 'Platform Information'
+    cmds:
+      - exec:
+        cmd:
+          - python
+          - -c
+          - |
+            import platform
+            print(f'system: {platform.system()}')
+            print(f'release: {platform.release()}')
+            print(f'machine: {platform.machine()}')
+        output: platform.inf

tools/idf_py_actions/diag/recipes/manager.yml

@@ -0,0 +1,20 @@
+description: IDF Component Manager information
+tags: [manager, base, project]
+output: manager
+steps:
+  - name: 'IDF Component Manager'
+    cmds:
+      - exec:
+        cmd: 'python -m idf_component_manager version'
+        output: manager.ver
+      - file:
+        path: '${PROJECT_DIR}/dependencies.lock'
+      - glob:
+        # Gather all idf_component.yml files from the project directory and
+        # save them in directories relative to the project directory within
+        # the idf_component directory.
+        pattern: 'idf_component.yml'
+        recursive: true
+        relative: true
+        path: '${PROJECT_DIR}'
+        output: 'idf_component/'

tools/idf_py_actions/diag/recipes/project.yml

@@ -0,0 +1,94 @@
+description: ESP-IDF project information and artifacts
+tags: [project]
+output: project
+steps:
+  - name: 'Project Description'
+    cmds:
+      - file:
+        path: '${BUILD_DIR}/project_description.json'
+
+  - name: 'Compile Commands'
+    cmds:
+      - file:
+        path: '${BUILD_DIR}/compile_commands.json'
+
+  - name: 'Linker Scripts'
+    cmds:
+      - file:
+        path: '${BUILD_DIR}/esp-idf/esp_system/ld/memory.ld'
+      - file:
+        path: '${BUILD_DIR}/esp-idf/esp_system/ld/sections.ld'
+
+  - name: 'Flash Arguments'
+    cmds:
+      - file:
+        path: '${BUILD_DIR}/flash_app_args'
+      - file:
+        path: '${BUILD_DIR}/flash_args'
+      - file:
+        path: '${BUILD_DIR}/bootloader-flash_args'
+      - file:
+        path: '${BUILD_DIR}/flash_bootloader_args'
+      - file:
+        path: '${BUILD_DIR}/flasher_args.json'
+      - file:
+        path: '${BUILD_DIR}/flash_project_args'
+      - file:
+        path: '${BUILD_DIR}/partition-table-flash_args'
+
+  - name: 'Sdkconfig'
+    cmds:
+      - file:
+        path: '${PROJECT_DIR}/sdkconfig'
+
+  - name: 'Logs'
+    output: 'logs'
+    cmds:
+      # Build logs
+      - glob:
+        pattern: 'idf_py_stdout_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: ninja all$'
+        output: 'build/'
+        mtime: True
+      - glob:
+        pattern: 'idf_py_stderr_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: ninja all$'
+        output: 'build/'
+        mtime: True
+
+      # Flash logs
+      - glob:
+        pattern: 'idf_py_stdout_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: ninja flash$'
+        output: 'flash/'
+        mtime: True
+      - glob:
+        pattern: 'idf_py_stderr_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: ninja flash$'
+        output: 'flash/'
+        mtime: True
+
+      # Monitor logs
+      - glob:
+        pattern: 'idf_py_stdout_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: .*idf_monitor.py.*$'
+        output: 'monitor/'
+        mtime: True
+      - glob:
+        pattern: 'idf_py_stderr_output*'
+        path: '${BUILD_DIR}/log'
+        regex: '^Command: .*idf_monitor.py.*$'
+        output: 'monitor/'
+        mtime: True
+
+
+  - name: 'Link Map File'
+    cmds:
+      - glob:
+        pattern: '*.map'
+        path: '${BUILD_DIR}'

tools/idf_py_actions/diag/recipes/python.yml

@@ -0,0 +1,18 @@
+description: ESP-IDF python and virtual environment information
+tags: [python, base, project]
+output: python
+steps:
+  - name: 'Python Version'
+    cmds:
+      - exec:
+        cmd: 'python --version'
+        output: python.ver
+
+  - name: 'Python Virtual Environment Packages'
+    cmds:
+      - exec:
+        cmd: 'python -m pip freeze'
+        output: pip.freeze
+      - exec:
+        cmd: 'python -m pip list'
+        output: pip.list

tools/idf_py_actions/diag/recipes/tools.yml

@@ -0,0 +1,12 @@
+description: ESP-IDF tools information
+tags: [tools, base, project]
+output: tools
+steps:
+  - name: 'ESP-IDF Tools Info'
+    cmds:
+      - exec:
+        cmd: 'python ${IDF_PATH}/tools/idf_tools.py check'
+        output: tools.check
+      - exec:
+        cmd: 'python ${IDF_PATH}/tools/idf_tools.py list'
+        output: tools.list

tools/test_idf_diag/pytest.ini

@@ -0,0 +1,12 @@
+[pytest]
+addopts = -s -p no:pytest_embedded
+
+# log related
+log_cli = True
+log_cli_level = INFO
+log_cli_format = %(asctime)s %(levelname)s %(message)s
+log_cli_date_format = %Y-%m-%d %H:%M:%S
+
+## log all to `system-out` when case fail
+junit_logging = stdout
+junit_log_passing_tests = False

tools/test_idf_diag/test_idf_diag.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+# SPDX-FileCopyrightText: 2024 Espressif Systems (Shanghai) CO LTD
+# SPDX-License-Identifier: Apache-2.0
+import logging
+import os
+import sys
+from pathlib import Path
+from shutil import copytree
+from subprocess import PIPE
+from subprocess import run
+from subprocess import STDOUT
+from tempfile import TemporaryDirectory
+from typing import Any
+from typing import Optional
+from typing import Tuple
+from typing import Union
+
+
+IDF_PATH = Path(os.environ['IDF_PATH'])
+IDF_PY_PATH = IDF_PATH / 'tools' / 'idf.py'
+IDF_DIAG_PY_PATH = IDF_PATH / 'tools' / 'idf_diag.py'
+HELLO_WORLD_PATH = IDF_PATH / 'examples' / 'get-started' / 'hello_world'
+
+PathLike = Union[str, Path]
+
+
+def run_cmd(*cmd: PathLike, cwd: Optional[PathLike]=None, check: bool=True, text: bool=True) -> Tuple[int, str]:
+    logging.info('running: {}'.format(' '.join([str(arg) for arg in cmd])))
+    p = run(cmd, stdout=PIPE, stderr=STDOUT, cwd=cwd, check=check, text=text)
+    return p.returncode, p.stdout
+
+
+def run_idf_py(*args: PathLike, **kwargs: Any) -> Tuple[int, str]:
+    return run_cmd(sys.executable, IDF_PY_PATH, *args, **kwargs)
+
+
+def test_idf_diag() -> None:
+    # Basic test, compile the hello_world example, generate a report directory, and archive it.
+
+    # temporary directories
+    tmpdir = TemporaryDirectory()
+    app_path = Path(tmpdir.name) / 'app'
+    report_path = Path(tmpdir.name) / 'report'
+
+    # build hello world example
+    logging.info('building testing hello_world example')
+    copytree(HELLO_WORLD_PATH, app_path)
+    run_idf_py('fullclean', cwd=app_path)
+    run_idf_py('build', cwd=app_path)
+
+    # create report
+    logging.info('creating report')
+    run_idf_py('diag', '--output', report_path, cwd=app_path)
+
+    # archive report
+    logging.info('creating report archive')
+    run_idf_py('diag', '--zip', report_path)
+
+    # list recipes
+    logging.info('list recipes')
+    run_idf_py('diag', '--list')
+
+    # check recipes
+    logging.info('check recipes')
+    run_idf_py('diag', '--check')
+
+    # check redaction
+    logging.info('check redaction')
+    idf_component_path = app_path / 'idf_component.yml'
+    idf_component_path.write_text('https://username:password@github.com/username/repository.git')
+    run_idf_py('diag', '--force', '--output', report_path, cwd=app_path)
+    idf_component_path.unlink()
+    with open(report_path / 'manager' / 'idf_component' / 'idf_component.yml', 'r') as f:
+        data = f.read()
+        assert 'https://[REDACTED]@github.com/username/repository.git' in data