blackbeard420/esp-idf
1
0
Fork 0
mirror of https://github.com/espressif/esp-idf synced 2025-03-08 23:59:08 -05:00
Code Issues Projects Releases Wiki Activity

feat(diag): use the esp-idf-diag module as a backend

Browse Source 
Replace the original diag implementation in idf.py with the new one in a
separate esp-idf-diag python package. The interface is kept the same as
it was. The simple idf.py diag test is also preserved.

Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit is contained in:
Frantisek Hrbata 2025-02-10 16:50:41 +01:00
parent f428a1e784
commit bdc94935fa
13 changed files with 90 additions and 1651 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
tools/idf_py_actions/diag/purge/README.md
View File

@ -1,16 +0,0 @@
# Purge format description for idf.py diag
Once diagnostic information is collected, the purge file is utilized to remove
any sensitive data from the gathered files. By default, the purge file located
at `tools/idf_py_actions/diag/purge/purge.yml` is used unless it is specified
with the --purge argument, in which case the default file is not used.
## Overview of Purge Structure
It is a straightforward YAML file that includes a list of regular expressions
and the corresponding strings that should replace any matches.
- regex: regular expression to look for
repl: substitute string for match
The `regex.sub` function from Python is used internally.

3
tools/idf_py_actions/diag/purge/purge.yml
View File

@ -1,3 +0,0 @@
# Remove user, password, token from URL
- regex: '://[^@]+@'
repl: '://[REDACTED]@'

293
tools/idf_py_actions/diag/recipes/README.md
View File

@ -1,293 +0,0 @@
# Recipe format description for idf.py diag
The `idf.py diag` command processes one or more `recipe` files. Each `recipe`
file outlines a collection of related data and files that should be gathered.
For instance, the `idf.yml` recipe gathers information related to the ESP-IDF.
`Recipes` are formatted in YAML. The output from each `recipe` consists of a
set of files located in the diagnostic report directory, which may be copied or
generated as a result of executing certain commands.
A `recipe` is made up of one or more `steps`, and each `step` contains one or
more commands. A recipe can consist of just one `step` where all the commands
are carried out. The aim is to split the `recipe` into logical steps if it is
convenient. For instance, a `project.yml` `recipe` might include one `step` to
gather all log files and another to collect linker script files, although these
tasks could also be completed in a single `step`. Refer to the `recipes` in
this directory for examples.
## Overview of Recipe Structure
description: brief recipe description
tags: list of tags that can be used to identify the recipe
output: root recipe directory for its output in the report folder
steps: list of steps to execute within a recipe
- name: brief step description
output: step directory for its output in the report folder
cmds: list of commands to execute within a step
- cmd: command name
arg: command argument
...
...
...
## Recipe variables
The `recipe` can utilize the following variables. The `idf.py diag` assigns
values to these variables and expands them in the recipe upon loading. To use a
variable, format it as `${NAME}`, such as `${IDF_PATH}`.
* PROJECT_DIR
Project directory specified by idf.py using the `-C` or `--project-dir`
option.
* BUILD_DIR
Build directory specified by idf.py using the `-B` or `--build-dir` option.
* IDF_PATH
IDF path as defined in the environment variable.
* REPORT_DIR
The report directory is where all the recipe outputs are stored. Keep in
mind that during the execution of `idf.py diag`, it points to a temporary
directory, which is moved to its final destination once the report is
successfully completed.
## Recipe
* description: string (required)
Short `recipe` description, which is shown in `idf.py diag` progress.
* tags: list (optional)
Strings which identify this `recipe`. Used to specify which `recipes`
should the `idf.py diag` use. All `recipes` with a given tag are used.
* output: string (optional)
Global output directory for the `recipe`. This directory serves as the main
directory for all files produced by this `recipe` within the report
directory. For instance, if it is set to `idf` and the report directory is
`report`, then all files collected by this `recipe` will be stored in the
`report/idf` directory. This helps organize the collected files within the
report directory.
* steps: list (required)
One or more `steps` to follow for this `recipe`. This allows, but does not
require, splitting the `recipe` into more logical sections. For example,
one `step` could gather all log files, while another might collect
environment information.
## Step
* name: string (required)
Brief description of the `step`, displayed in the `idf.py diag` progress
beneath the `recipe` description.
* system: string (optional)
Can be used to restrict the operating system on which the step will be
executed. It should be Linux, Darwin, or Windows. If specified, the step
will only run on the designated system.
* output: string (optional)
Global output directory for the `step`. This directory serves as the main
directory for all files produced by this `step` within the recipe `output`
directory. For instance, if it is set to `logs` and the report directory
is `report` and recipe `output` is `idf`, then all files collected by this
`step` will be stored in the `report/idf/logs` directory.
* cmds: list (required)
Sequence of commands to be executed in this `step`.
## Commands
The following commands can be used within a `step` in the `recipe`. Each
command consists of a list that includes the command name, such as `exec` or
`glob`, along with its arguments. Please be aware that if a command fails, it
does not terminate `idf.py diag`; all commands will still be executed. The
command mapping key has no value, and if it is present, it is ignored.
### file
Copy the specified file from the given path to the report directory. If no
`output` argument is provided, the file is copied using its original name.
* path: string (required)
Path to the source file.
* output: string (optional)
The destination path for the file within the report directory. If it ends
with a `/` character, it is treated as a directory, and the source file
name from the `path` argument is added to it. Otherwise, it is considered a
complete path including the file name. If not provided, the source file
name is used as the destination file name. The complete destination path in
the report directory is built starting with the `output` directory
specified in the `recipe`, followed by the `output` directory specified in
the`step`, and finally the `output` specified in the `file` command, if
each is provided. All directories that do not exist in the output path are
created.
Example:
- file:
path: '${BUILD_DIR}/compile_commands.json'
output: 'build/json_files/commands.json'
### exec
Run the given command and save its output.
* cmd: string or list (required)
Command to run. If it's a string, it starts via the shell; if it's a list,
it starts without shell expansions. Using the list format can help avoid
escaping command arguments.
* output: string (optional)
The path in the report directory where the command's standard output should
be stored. If not specified, the command output will not be saved. The
complete destination path in the report directory is built starting with
the `output` directory specified in the `recipe`, followed by the `output`
directory specified in the `step`, and finally the `output` specified in
the `exec` command, if each is provided. The `/` character at the end is
disregarded in contrast to the `file` command.
* stderr: string (optional)
Similar to `output`, but specifically for standard error output.
* timeout: int (optional)
The number of seconds to wait for the command to execute.
* append: boolean (optional)
Append the command's standard output and standard error to the files
specified by the `output` and `stderr` arguments.
Example:
- exec:
cmd: 'idf.py --version'
timeout: 10
output: esp_idf.ver
- exec:
cmd:
- python
- -c
- |
import platform
print(f'system: {platform.system()}')
### env
Store details of the specified environment variables in a file using the format
variable=value.
* vars: list (required)
A list of names for environment variables to gather.
* regex: string (optional)
Optional regular expression to gather environment variables that match it.
* output: string (optional)
The path in the report directory where the environment variable information
should be stored. If not specified, nothing will be saved. The complete
destination path in the report directory is built starting with the
`output` directory specified in the `recipe`, followed by the `output`
directory specified in the `step`, and finally the `output` specified in
the `env` command, if each is provided. The `/` character at the end is
disregarded, in contrast to the `file` command.
* append: boolean (optional)
Append the environmental variables information to the file specified by the
`output` argument.
Example:
- env:
vars:
- IDF_PATH
- IDF_PYTHON_ENV_PATH
- IDF_TOOLS_PATH
- PATH
- OPENOCD_SCRIPTS
- PYTHONPATH
- MSYSTEM
regex: '.*IDF.*|.*ESP.*'
output: environment.var
### glob
Find pathnames that match a specific pattern within a specified directory, and
include them in the report directory.
* pattern: string (required)
Pattern matching with wildcards.
* path: string (required)
Directory to look for files that match the `pattern`.
* output: string (optional)
Every file that matches the `pattern` is stored in the report directory at
the location specified by `output`. If `output` ends with a `/`, it is
interpreted as a directory, and the matched file name is appended to it. If
it does not end with a `/`, it is regarded as a full path including the
file name. If `output` is not specified, the name of the matching file is
used as the destination file name.
The full destination path in the report directory for each matched file
is constructed by using the `output` directory from the `recipe`, followed
by the `output` directory from the `step`, and finally the `output` from
the `glob` command, if each is available. Any directories that do not exist
in the output path are created. If the `relative` argument is set and the
`output` argument in the `glob` command ends with `/`, the file's relative
path to the `path` argument is appended to the `output` argument in the
`glob` command. A `.<cnt>` suffix is added to a file path if that path
already exists in the report directory. The `<cnt>` is a number that
increases until a unique path is found in the report directory.
* mtime: boolean (optional)
If multiple files are found, add only the one that was modified most recently.
* recursive: boolean (optional)
Recursively search the `path`.
* relative: boolean (optional)
Save the file in the `output` directory, preserving the same relative path
it has in the `path` directory.
* regex: string (optional)
Only add files whose content matches the specified regular expression.
Example:
- glob:
pattern: 'idf_py_stdout_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: .*idf_monitor.py.*$'
output: 'logs/monitor/'
mtime: True

23
tools/idf_py_actions/diag/recipes/environment.yml
View File

@ -1,23 +0,0 @@
description: Shell environment information
tags: [environment, base, project]
output: environment
steps:
- name: 'Enviromental Variables'
cmds:
- env:
vars:
- IDF_PATH
- IDF_PYTHON_ENV_PATH
- IDF_TOOLS_PATH
- PATH
- OPENOCD_SCRIPTS
- PYTHONPATH
- MSYSTEM
regex: '.*IDF.*|.*ESP.*'
output: environment.var
- name: 'Python Environmental Variables'
cmds:
- env:
regex: '.*PYTHON.*'
output: python.var

15
tools/idf_py_actions/diag/recipes/idf.yml
View File

@ -1,15 +0,0 @@
description: Basic ESP-IDF information
tags: [idf, base, project]
output: idf
steps:
- name: 'ESP-IDF Version'
cmds:
- exec:
cmd: 'idf.py --version'
output: esp_idf.ver
- name: 'ESP-IDF Git Version'
cmds:
- exec:
cmd: 'git -C ${IDF_PATH} describe'
output: esp_idf_git.ver

20
tools/idf_py_actions/diag/recipes/manager.yml
View File

@ -1,20 +0,0 @@
description: IDF Component Manager information
tags: [manager, base, project]
output: manager
steps:
- name: 'IDF Component Manager'
cmds:
- exec:
cmd: 'python -m idf_component_manager version'
output: manager.ver
- file:
path: '${PROJECT_DIR}/dependencies.lock'
- glob:
# Gather all idf_component.yml files from the project directory and
# save them in directories relative to the project directory within
# the idf_component directory.
pattern: 'idf_component.yml'
recursive: true
relative: true
path: '${PROJECT_DIR}'
output: 'idf_component/'

94
tools/idf_py_actions/diag/recipes/project.yml
View File

@ -1,94 +0,0 @@
description: ESP-IDF project information and artifacts
tags: [project]
output: project
steps:
- name: 'Project Description'
cmds:
- file:
path: '${BUILD_DIR}/project_description.json'
- name: 'Compile Commands'
cmds:
- file:
path: '${BUILD_DIR}/compile_commands.json'
- name: 'Linker Scripts'
cmds:
- file:
path: '${BUILD_DIR}/esp-idf/esp_system/ld/memory.ld'
- file:
path: '${BUILD_DIR}/esp-idf/esp_system/ld/sections.ld'
- name: 'Flash Arguments'
cmds:
- file:
path: '${BUILD_DIR}/flash_app_args'
- file:
path: '${BUILD_DIR}/flash_args'
- file:
path: '${BUILD_DIR}/bootloader-flash_args'
- file:
path: '${BUILD_DIR}/flash_bootloader_args'
- file:
path: '${BUILD_DIR}/flasher_args.json'
- file:
path: '${BUILD_DIR}/flash_project_args'
- file:
path: '${BUILD_DIR}/partition-table-flash_args'
- name: 'Sdkconfig'
cmds:
- file:
path: '${PROJECT_DIR}/sdkconfig'
- name: 'Logs'
output: 'logs'
cmds:
# Build logs
- glob:
pattern: 'idf_py_stdout_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: ninja all$'
output: 'build/'
mtime: True
- glob:
pattern: 'idf_py_stderr_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: ninja all$'
output: 'build/'
mtime: True
# Flash logs
- glob:
pattern: 'idf_py_stdout_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: ninja flash$'
output: 'flash/'
mtime: True
- glob:
pattern: 'idf_py_stderr_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: ninja flash$'
output: 'flash/'
mtime: True
# Monitor logs
- glob:
pattern: 'idf_py_stdout_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: .*idf_monitor.py.*$'
output: 'monitor/'
mtime: True
- glob:
pattern: 'idf_py_stderr_output*'
path: '${BUILD_DIR}/log'
regex: '^Command: .*idf_monitor.py.*$'
output: 'monitor/'
mtime: True
- name: 'Link Map File'
cmds:
- glob:
pattern: '*.map'
path: '${BUILD_DIR}'

18
tools/idf_py_actions/diag/recipes/python.yml
View File

@ -1,18 +0,0 @@
description: ESP-IDF python and virtual environment information
tags: [python, base, project]
output: python
steps:
- name: 'Python Version'
cmds:
- exec:
cmd: 'python --version'
output: python.ver
- name: 'Python Virtual Environment Packages'
cmds:
- exec:
cmd: 'python -m pip freeze'
output: pip.freeze
- exec:
cmd: 'python -m pip list'
output: pip.list

49
tools/idf_py_actions/diag/recipes/system.yml
View File

@ -1,49 +0,0 @@
description: System information
tags: [system, base, project]
steps:
- name: 'Platform Information'
cmds:
- exec:
cmd:
- python
- -c
- |
import platform
print(f'system: {platform.system()}')
print(f'release: {platform.release()}')
print(f'machine: {platform.machine()}')
output: platform.inf
- name: 'Linux System Information'
system: Linux
output: linux
cmds:
- file:
path: '/proc/cpuinfo'
- file:
path: '/etc/os-release'
- exec:
cmd: 'uname -a'
output: uname
- name: 'Darwin Information'
system: Darwin
output: darwin
cmds:
- exec:
cmd: 'sysctl machdep.cpu'
output: machdep.cpu
- exec:
cmd: 'sw_vers'
output: sw_vers
- exec:
cmd: 'uname -a'
output: uname
- name: 'Windows Information'
system: Windows
output: windows
cmds:
- exec:
cmd: 'systeminfo'
output: systeminfo

12
tools/idf_py_actions/diag/recipes/tools.yml
View File

@ -1,12 +0,0 @@
description: ESP-IDF tools information
tags: [tools, base, project]
output: tools
steps:
- name: 'ESP-IDF Tools Info'
cmds:
- exec:
cmd: 'python ${IDF_PATH}/tools/idf_tools.py check'
output: tools.check
- exec:
cmd: 'python ${IDF_PATH}/tools/idf_tools.py list'
output: tools.list

1195
tools/idf_py_actions/diag_ext.py
View File

File diff suppressed because it is too large Load Diff

1
tools/requirements/requirements.core.txt
View File

@ -20,6 +20,7 @@ esp-idf-kconfig
esp-idf-monitor
esp-idf-nvs-partition-gen
esp-idf-size
esp-idf-diag
esp-idf-panic-decoder
pyclang
construct

2
tools/test_idf_diag/test_idf_diag.py
View File

@ -72,4 +72,4 @@ def test_idf_diag() -> None:
idf_component_path.unlink()
with open(report_path / 'manager' / 'idf_component' / 'idf_component.yml', 'r') as f:
data = f.read()
assert 'https://[REDACTED]@github.com/username/repository.git' in data
assert 'https://[XXX]@github.com/username/repository.git' in data