tools: added idf.py coredump sub-command

This commit is contained in:
Aleksei Apaseev 2022-09-27 10:58:34 +08:00 committed by Aleksei Apaseev
parent 72a9bbebd3
commit 06ac2166d6
4 changed files with 208 additions and 109 deletions

View File

@ -9,10 +9,10 @@ Upon the crash system enters panic state, prints some information and halts or r
the reason of failure on PC later on. Core dump contains snapshots of all tasks in the system at the moment of failure. Snapshots include tasks control blocks (TCB) and stacks.
So it is possible to find out what task, at what instruction (line of code) and what callstack of that task lead to the crash. It is also possible dumping variables content on
demand if previously attributed accordingly.
ESP-IDF provides special script `espcoredump.py` to help users to retrieve and analyse core dumps. This tool provides two commands for core dumps analysis:
ESP-IDF provides special commands to help users to retrieve and analyse core dumps:
* ``info_corefile`` - prints crashed task's registers, callstack, list of available tasks in the system, memory regions and contents of memory stored in core dump (TCBs and stacks)
* ``dbg_corefile`` - creates core dump ELF file and runs GDB debug session with this file. User can examine memory, variables and tasks states manually. Note that since not all memory is saved in core dump only values of variables allocated on stack will be meaningful
* ``idf.py coredump-info`` - prints crashed task's registers, callstack, list of available tasks in the system, memory regions and contents of memory stored in core dump (TCBs and stacks)
* ``idf.py coredump-debug`` - creates core dump ELF file and runs GDB debug session with this file. User can examine memory, variables and tasks states manually. Note that since not all memory is saved in core dump only values of variables allocated on stack will be meaningful
For more information about core dump internals see the - :doc:`Core dump internals <core_dump_internals>`
@ -92,13 +92,13 @@ The example of generic command to analyze core dump from flash is:
.. code-block:: bash
espcoredump.py -p </path/to/serial/port> info_corefile </path/to/program/elf/file>
idf.py coredump-info
or
.. code-block:: bash
espcoredump.py -p </path/to/serial/port> dbg_corefile </path/to/program/elf/file>
idf.py coredump-debug
Print core dump to UART
-----------------------
@ -108,13 +108,13 @@ then run the following command:
.. code-block:: bash
espcoredump.py --chip {IDF_TARGET_PATH_NAME} info_corefile -t b64 -c </path/to/saved/base64/text> </path/to/program/elf/file>
idf.py coredump-info -c </path/to/saved/base64/text>
or
.. code-block:: bash
espcoredump.py --chip {IDF_TARGET_PATH_NAME} dbg_corefile -t b64 -c </path/to/saved/base64/text> </path/to/program/elf/file>
idf.py coredump-debug -c </path/to/saved/base64/text>
Base64-encoded body of core dump will be between the following header and footer::
@ -130,7 +130,7 @@ ROM Functions in Backtraces
It is possible situation that at the moment of crash some tasks or/and crashed task itself have one or more ROM functions in their callstacks.
Since ROM is not part of the program ELF it will be impossible for GDB to parse such callstacks, because it tries to analyse functions' prologues to accomplish that.
In that case callstack printing will be broken with error message at the first ROM function.
To overcome this issue, you can use the `ROM ELF <https://github.com/espressif/esp-rom-elfs/releases>`_ provided by Espressif. You can find the {IDF_TARGET_PATH_NAME}'s corresponding ROM ELF file from the list of released archives. The ROM ELF file can then be passed to ``espcoredump.py``. More details about ROM ELFs can be found `here <https://github.com/espressif/esp-rom-elfs/blob/master/README.md>`_.
To overcome this issue, `ROM ELF <https://github.com/espressif/esp-rom-elfs/releases>`_ provided by Espressif is loaded automatically based on the target and its revision. More details about ROM ELFs can be found `here <https://github.com/espressif/esp-rom-elfs/blob/master/README.md>`_.
Dumping variables on demand
---------------------------
@ -172,7 +172,7 @@ Example
.. code-block:: bash
espcoredump.py -p PORT dbg_corefile <path/to/elf>
idf.py coredump-debug
6. In GDB shell, type ``p global_var`` to get the variable content:
@ -181,52 +181,10 @@ Example
(gdb) p global_var
$1 = 25 '\031'
Running ``espcoredump.py``
--------------------------
Running ``idf.py coredump-info`` and ``idf.py coredump-debug``
--------------------------------------------------------------
Generic command syntax: ``espcoredump.py [options] command [args]``
:Script Options:
--chip {auto,esp32,esp32s2,esp32s3,esp32c2,esp32c3}
Target chip type. Default value is "auto"
--port PORT, -p PORT Serial port device. Either "chip" or "port" need to be specified to determine the port when you have multi-target connected at the same time.
--baud BAUD, -b BAUD Serial port baud rate used when flashing/reading
--gdb-timeout-sec GDB_TIMEOUT_SEC
Overwrite the default internal delay for gdb responses
:Commands:
**dbg_corefile** Starts GDB debugging session with specified corefile
**info_corefile** Print core dump info from file
:Command Arguments:
--debug DEBUG, -d DEBUG
Log level (0..3)
--gdb GDB, -g GDB Path to gdb
--core CORE, -c CORE Path to core dump file (if skipped core dump will be read from flash)
--core-format {b64,elf,raw}, -t {b64,elf,raw}
File specified with "-c" is an ELF ("elf"), raw (raw) or base64-encoded (b64) binary
--off OFF, -o OFF Offset of coredump partition in flash (type "idf.py partition-table" to see).
--save-core SAVE_CORE, -s SAVE_CORE
Save core to file. Otherwise temporary core file will be deleted. Does not work with "-c"
--rom-elf ROM_ELF, -r ROM_ELF
Path to ROM ELF file. Will use "<target>_rom.elf" if not specified
--print-mem, -m Print memory dump. Only valid when info_corefile.
**<prog>** Path to program ELF file.
``idf.py coredump-info --help`` and ``idf.py coredump-debug --help`` commands can be used to get more details on usage
Related Documents
^^^^^^^^^^^^^^^^^

View File

@ -9,14 +9,18 @@ import subprocess
import sys
import threading
import time
from base64 import b64decode
from textwrap import indent
from threading import Thread
from typing import Any, Dict, List, Optional
from click import INT
from click.core import Context
from esp_coredump import CoreDump
from idf_py_actions.constants import OPENOCD_TAGET_CONFIG, OPENOCD_TAGET_CONFIG_DEFAULT
from idf_py_actions.errors import FatalError
from idf_py_actions.tools import PropertyDict, ensure_build_directory
from idf_py_actions.serial_ext import BAUD_RATE, PORT
from idf_py_actions.tools import PropertyDict, ensure_build_directory, get_default_serial_port, get_sdkconfig_value
PYTHON = sys.executable
ESP_ROM_INFO_FILE = 'roms.json'
@ -132,13 +136,94 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
print('Failed to close/kill {}'.format(target))
processes[target] = None # to indicate this has ended
def _get_espcoredump_instance(ctx: Context,
args: PropertyDict,
gdb_timeout_sec: int = None,
core: str = None,
save_core: str = None) -> CoreDump:
ensure_build_directory(args, ctx.info_name)
project_desc = get_project_desc(args, ctx)
coredump_to_flash_config = get_sdkconfig_value(project_desc['config_file'],
'CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH')
coredump_to_flash = coredump_to_flash_config.rstrip().endswith('y') if coredump_to_flash_config else False
prog = os.path.join(project_desc['build_dir'], project_desc['app_elf'])
esp_port = args.port or get_default_serial_port()
espcoredump_kwargs = dict()
espcoredump_kwargs['port'] = esp_port
espcoredump_kwargs['baud'] = args.baud
espcoredump_kwargs['gdb_timeout_sec'] = gdb_timeout_sec
# for reproducible builds
extra_gdbinit_file = project_desc.get('debug_prefix_map_gdbinit', None)
if extra_gdbinit_file:
espcoredump_kwargs['extra_gdbinit_file'] = extra_gdbinit_file
core_format = None
if core:
espcoredump_kwargs['core'] = core
espcoredump_kwargs['chip'] = get_sdkconfig_value(project_desc['config_file'], 'CONFIG_IDF_TARGET')
core_format = get_core_file_format(core)
elif coredump_to_flash:
# If the core dump is read from flash, we don't need to specify the --core-format argument at all.
# The format will be determined automatically
pass
else:
print('Path to core dump file is not provided. '
"Core dump can't be read from flash since this option is not enabled in menuconfig")
sys.exit(1)
if core_format:
espcoredump_kwargs['core_format'] = core_format
if save_core:
espcoredump_kwargs['save_core'] = save_core
espcoredump_kwargs['prog'] = prog
return CoreDump(**espcoredump_kwargs)
def get_core_file_format(core_file: str) -> str:
bin_v1 = 1
bin_v2 = 2
elf_crc32 = 256
elf_sha256 = 257
with open(core_file, 'rb') as f:
coredump_bytes = f.read(16)
if coredump_bytes.startswith(b'\x7fELF'):
return 'elf'
core_version = int.from_bytes(coredump_bytes[4:7], 'little')
if core_version in [bin_v1, bin_v2, elf_crc32, elf_sha256]:
# esp-coredump will determine automatically the core format (ELF or BIN)
return 'raw'
with open(core_file) as c:
coredump_str = c.read()
try:
b64decode(coredump_str)
except Exception:
print('The format of the provided core-file is not recognized. '
'Please ensure that the core-format matches one of the following: ELF (“elf”), '
'raw (raw) or base64-encoded (b64) binary')
sys.exit(1)
else:
return 'b64'
def is_gdb_with_python(gdb: str) -> bool:
# execute simple python command to check is it supported
return subprocess.run([gdb, '--batch-silent', '--ex', 'python import os'], stderr=subprocess.DEVNULL).returncode == 0
return subprocess.run([gdb, '--batch-silent', '--ex', 'python import os'],
stderr=subprocess.DEVNULL).returncode == 0
def get_normalized_path(path: str) -> str:
if os.name == 'nt':
return os.path.normpath(path).replace('\\','\\\\')
return os.path.normpath(path).replace('\\', '\\\\')
return path
def get_rom_if_condition_str(date_addr: int, date_str: str) -> str:
@ -286,7 +371,8 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
project_desc = json.load(f)
return project_desc
def openocd(action: str, ctx: Context, args: PropertyDict, openocd_scripts: Optional[str], openocd_commands: str) -> None:
def openocd(action: str, ctx: Context, args: PropertyDict, openocd_scripts: Optional[str],
openocd_commands: str) -> None:
"""
Execute openocd as external tool
"""
@ -311,7 +397,8 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
process = subprocess.Popen(args, stdout=openocd_out, stderr=subprocess.STDOUT, bufsize=1)
except Exception as e:
print(e)
raise FatalError('Error starting openocd. Please make sure it is installed and is present in executable paths', ctx)
raise FatalError(
'Error starting openocd. Please make sure it is installed and is present in executable paths', ctx)
processes['openocd'] = process
processes['openocd_outfile'] = openocd_out
@ -326,7 +413,8 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
args.append('-ix={}'.format(debug_prefix_gdbinit))
return args
def gdbui(action: str, ctx: Context, args: PropertyDict, gdbgui_port: Optional[str], gdbinit: Optional[str], require_openocd: bool) -> None:
def gdbui(action: str, ctx: Context, args: PropertyDict, gdbgui_port: Optional[str], gdbinit: Optional[str],
require_openocd: bool) -> None:
"""
Asynchronous GDB-UI target
"""
@ -436,16 +524,51 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
# Valid scenario: watch_openocd task won't be in the list if openocd not started from idf.py
pass
def coredump_info(action: str,
ctx: Context,
args: PropertyDict,
gdb_timeout_sec: int,
core: str = None,
save_core: str = None) -> None:
espcoredump = _get_espcoredump_instance(ctx=ctx, args=args, gdb_timeout_sec=gdb_timeout_sec, core=core,
save_core=save_core)
espcoredump.info_corefile()
def coredump_debug(action: str,
ctx: Context,
args: PropertyDict,
core: str = None,
save_core: str = None) -> None:
espcoredump = _get_espcoredump_instance(ctx=ctx, args=args, core=core, save_core=save_core)
espcoredump.dbg_corefile()
coredump_base = [
{
'names': ['--core', '-c'],
'help': 'Path to core dump file (if skipped core dump will be read from flash)',
},
{
'names': ['--save-core', '-s'],
'help': 'Save core to file. Otherwise temporary core file will be deleted.',
},
]
gdb_timeout_sec_opt = {
'names': ['--gdb-timeout-sec'],
'type': INT,
'default': 1,
'help': 'Overwrite the default internal delay for gdb responses',
}
fail_if_openocd_failed = {
'names': ['--require-openocd', '--require_openocd'],
'help':
('Fail this target if openocd (this targets dependency) failed.\n'),
'help': 'Fail this target if openocd (this targets dependency) failed.\n',
'is_flag': True,
'default': False,
}
gdbinit = {
'names': ['--gdbinit'],
'help': ('Specify the name of gdbinit file to use\n'),
'help': 'Specify the name of gdbinit file to use\n',
'default': None,
}
debug_actions = {
@ -510,6 +633,19 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
'options': [gdbinit, fail_if_openocd_failed],
'order_dependencies': ['all', 'flash'],
},
'coredump-info': {
'callback': coredump_info,
'help': 'Print crashed tasks registers, callstack, list of available tasks in the system, '
'memory regions and contents of memory stored in core dump (TCBs and stacks)',
'options': coredump_base + [PORT, BAUD_RATE, gdb_timeout_sec_opt], # type: ignore
'order_dependencies': ['all', 'flash'],
},
'coredump-debug': {
'callback': coredump_debug,
'help': 'Create core dump ELF file and run GDB debug session with this file.',
'options': coredump_base + [PORT, BAUD_RATE], # type: ignore
'order_dependencies': ['all', 'flash'],
},
'post-debug': {
'callback': post_debug,
'help': 'Utility target to read the output of async debug action and stop them.',

View File

@ -8,13 +8,30 @@ from typing import Any, Dict, List
import click
from idf_monitor_base.output_helpers import yellow_print
from idf_py_actions.errors import FatalError, NoSerialPortFoundError
from idf_py_actions.global_options import global_options
from idf_py_actions.tools import PropertyDict, RunTool, ensure_build_directory, get_sdkconfig_value, run_target
from idf_py_actions.tools import (PropertyDict, RunTool, ensure_build_directory, get_default_serial_port,
get_sdkconfig_value, run_target)
PYTHON = sys.executable
BAUD_RATE = {
'names': ['-b', '--baud'],
'help': 'Baud rate for flashing. It can imply monitor baud rate as well if it hasn\'t been defined locally.',
'scope': 'global',
'envvar': 'ESPBAUD',
'default': 460800,
}
PORT = {
'names': ['-p', '--port'],
'help': 'Serial port.',
'scope': 'global',
'envvar': 'ESPPORT',
'default': None,
}
def action_extensions(base_actions: Dict, project_path: str) -> Dict:
def _get_project_desc(ctx: click.core.Context, args: PropertyDict) -> Any:
desc_path = os.path.join(args.build_dir, 'project_description.json')
@ -24,33 +41,11 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
project_desc = json.load(f)
return project_desc
def _get_default_serial_port(args: PropertyDict) -> Any:
# Import is done here in order to move it after the check_environment() ensured that pyserial has been installed
try:
import esptool
import serial.tools.list_ports
ports = list(sorted(p.device for p in serial.tools.list_ports.comports()))
# high baud rate could cause the failure of creation of the connection
esp = esptool.get_default_connected_device(serial_list=ports, port=None, connect_attempts=4,
initial_baud=115200)
if esp is None:
raise NoSerialPortFoundError(
"No serial ports found. Connect a device, or use '-p PORT' option to set a specific port.")
serial_port = esp.serial_port
esp._port.close()
return serial_port
except NoSerialPortFoundError:
raise
except Exception as e:
raise FatalError('An exception occurred during detection of the serial port: {}'.format(e))
def _get_esptool_args(args: PropertyDict) -> List:
esptool_path = os.path.join(os.environ['IDF_PATH'], 'components/esptool_py/esptool/esptool.py')
esptool_wrapper_path = os.environ.get('ESPTOOL_WRAPPER', '')
if args.port is None:
args.port = _get_default_serial_port(args)
args.port = get_default_serial_port()
result = [PYTHON]
if os.path.exists(esptool_wrapper_path):
result += [esptool_wrapper_path]
@ -100,7 +95,7 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
yellow_print(msg)
no_reset = False
esp_port = args.port or _get_default_serial_port(args)
esp_port = args.port or get_default_serial_port()
monitor_args += ['-p', esp_port]
baud = monitor_baud or os.getenv('IDF_MONITOR_BAUD') or os.getenv('MONITORBAUD')
@ -168,7 +163,7 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
yellow_print('skipping flash since running on linux...')
return
esp_port = args.port or _get_default_serial_port(args)
esp_port = args.port or get_default_serial_port()
run_target(action, args, {'ESPBAUD': str(args.baud), 'ESPPORT': esp_port})
def erase_flash(action: str, ctx: click.core.Context, args: PropertyDict) -> None:
@ -192,27 +187,11 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
Calls ensure_build_directory() which will run cmake to generate a build
directory (with the specified generator) as needed.
"""
args.port = args.port or _get_default_serial_port(args)
args.port = args.port or get_default_serial_port()
ensure_build_directory(args, ctx.info_name)
run_target(target_name, args, {'ESPBAUD': str(args.baud), 'ESPPORT': args.port})
baud_rate = {
'names': ['-b', '--baud'],
'help': 'Baud rate for flashing. It can imply monitor baud rate as well if it hasn\'t been defined locally.',
'scope': 'global',
'envvar': 'ESPBAUD',
'default': 460800,
}
port = {
'names': ['-p', '--port'],
'help': 'Serial port.',
'scope': 'global',
'envvar': 'ESPPORT',
'default': None,
}
BAUD_AND_PORT = [baud_rate, port]
BAUD_AND_PORT = [BAUD_RATE, PORT]
serial_actions = {
'global_action_callbacks': [global_callback],
'actions': {
@ -244,7 +223,7 @@ def action_extensions(base_actions: Dict, project_path: str) -> Dict:
'help':
'Display serial output.',
'options': [
port, {
PORT, {
'names': ['--print-filter', '--print_filter'],
'help':
('Filter monitor output. '

View File

@ -12,6 +12,7 @@ from typing import Any, Dict, Generator, List, Match, Optional, TextIO, Tuple, U
import click
import yaml
from idf_py_actions.errors import NoSerialPortFoundError
from .constants import GENERATORS
from .errors import FatalError
@ -88,6 +89,31 @@ def idf_version() -> Optional[str]:
return version
def get_default_serial_port() -> Any:
# Import is done here in order to move it after the check_environment()
# ensured that pyserial has been installed
try:
import esptool
import serial.tools.list_ports
ports = list(sorted(p.device for p in serial.tools.list_ports.comports()))
# high baud rate could cause the failure of creation of the connection
esp = esptool.get_default_connected_device(serial_list=ports, port=None, connect_attempts=4,
initial_baud=115200)
if esp is None:
raise NoSerialPortFoundError(
"No serial ports found. Connect a device, or use '-p PORT' option to set a specific port.")
serial_port = esp.serial_port
esp._port.close()
return serial_port
except NoSerialPortFoundError:
raise
except Exception as e:
raise FatalError('An exception occurred during detection of the serial port: {}'.format(e))
# function prints warning when autocompletion is not being performed
# set argument stream to sys.stderr for errors and exceptions
def print_warning(message: str, stream: TextIO=None) -> None: