extmod/asyncio: Add a native asyncio REPL - #49
Draft
andrewleech wants to merge 193 commits into
Draft
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## review/native_async_repl #49 +/- ##
============================================================
+ Coverage 98.50% 98.54% +0.03%
============================================================
Files 177 182 +5
Lines 22927 23332 +405
Branches 0 5 +5
============================================================
+ Hits 22584 22992 +408
+ Misses 343 339 -4
- Partials 0 1 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Code size report: |
andrewleech
force-pushed
the
native_async_repl
branch
3 times, most recently
from
June 18, 2026 06:43
e66b858 to
39c82bb
Compare
This commit adds support for the ESP32-H2 processor from Espressif in the form of two board definitions - ESP32_GENERIC_H2 and M5STACK_NANOH2. Completed testing on the M5StackH2: - tested GPIO via on board LED - tested ADC channels can be read - I2C & SPI instantiation - BLE packet scanner - run_tests.py Signed-off-by: Rafal Wadowski <rafal@wadowski.com.au>
Added pins for I2S. Added ESP32-H2 check to reduce number of SPI peripherals. Signed-off-by: Rafal Wadowski <rafal@wadowski.com.au>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Raises LookupError for not implemented error handlers. Improves repr() rendering for unicode. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Only accepts `utf-8`, `utf8` or `ascii` Fixes micropython#15849 Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Fixes: issue 3364 Fixes: issue 13084 Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Fixes Issue 17827 Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Prevent the test from failing by not testing known unsupported characters. These will be documented in a cpydiff test. Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
These are now printed as characters rather than escaped bytes. Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
andrewleech
force-pushed
the
native_async_repl
branch
2 times, most recently
from
July 8, 2026 01:36
2b262f0 to
1cbd5fb
Compare
Three tests are rewritten using unittest, so that they don't need to be run under CPython to get the expected output. Because running under CPython requires port 8000 to be available, which it may not (at least for tests in the extmod directory having a port available should not be a requirement). Signed-off-by: Damien George <damien@micropython.org>
Similar to the parent commit, this means CPython doesn't need to run to get the expected output, and means port 8000 doesn't need to be available. Signed-off-by: Damien George <damien@micropython.org>
MicroPython allows unicode to be disabled to reduce firmware size, in which case str objects are essentially the same as bytes objects when it comes to code points between 128 and 255. But that behaviour has subtle differences to when unicode is enabled, eg string lengths can be different. Since unicode strings are arguably an important feature of Python, and users would likely be tripped up with it disabled, this commit changes the default level at which unicode is enabled from "extra" down to "basic". Costs about +1600 bytes on ARM Cortex-M targets. Signed-off-by: Damien George <damien@micropython.org>
Requires a tweak in makeqstrdefs to not enable debug output from the preprocessor, as -g3 adds a lot of extra macro definitions - some of which are picked up by makeqstrdefs otherwise. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
This variant has the smallest application partition in flash. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
The board's on-board SPI flash chip footprint shares pins with the side gpio headers, but the README only mentioned this in general terms. Add an explicit "Pin notes" subsection that lists which pins are used by SPI1 / FLASH_* per variant, calls out the SPI3 conflict on v2.0, and documents the empirical PB4 behaviour on v3.1 reported in issue micropython#18432. Fixes micropython#18432. Signed-off-by: Andrii Anoshyn <anoshyn.andrii@gmail.com>
Signed-off-by: Dryw Wade <dryw.wade@sparkfun.com>
Set a flag in the link-up transition and MAC-reconfig branches of eth_phy_link_status_poll(), then call eth_dhcp_restart_if_needed() once at the end of the function guarded by the flag. Avoids the back-to-back stop/start that occurred when both branches ran in the same poll iteration. Reported by @dpgeorge in the PR review of the original hot-plug support. Signed-off-by: Andrew Leech <andrew@alelec.net>
The transition block just above unconditionally assigns self->last_link_status = current_link_status, and the surrounding if condition already tested current_link_status true. So by the time this check runs, last_link_status is always true here and the early return can never fire. Signed-off-by: Andrew Leech <andrew@alelec.net>
Fixes micropython#19505. A regression was introduced in e48b985, where the USB sleep clock got enabled in stm32/main.c instead of the shared USB code, so mboot/main.c no longer enabled the USB sleep clock, causing failed enumeration. Signed-off-by: Dryw Wade <dryw.wade@sparkfun.com>
BUTTON was renamed to SW in 64c7045. Signed-off-by: Damien George <damien@micropython.org>
This allows devices to support partial update of the ROM segments. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This uses the new GET_MIN_PREPARE ioctl to support incremental preparation of the ROM partition. On very large partitions the preparation can take a long time (eg due to the erase time of many blocks) and this can lead to a timeout of mpremote. Using incremental preparation prevents the timeout and gives better feedback to the user as to the progress. Signed-off-by: Damien George <damien@micropython.org>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: jaenrig-ifx <enriquezgarcia.external@infineon.com>
Signed-off-by: jaenrig-ifx <enriquezgarcia.external@infineon.com>
external_call_depth is only read back (to detect the top-level call
boundary) when MICROPY_GC_SPLIT_HEAP_AUTO is enabled. The "standard"
variant does not enable this option, so its build was failing with:
main.c:50:15: error: variable 'external_call_depth' set but not
used [-Werror,-Wunused-but-set-global]
Fix by guarding the variable's declaration, increment, and decrement
all under the same #if, instead of leaving the decrement outside the
guard while the increment and declaration were inside/outside
inconsistently.
Fixes regression introduced by cdaf2de.
Signed-off-by: Harris <harrisjan06@gmail.com>
andrewleech
force-pushed
the
native_async_repl
branch
from
August 6, 2026 09:06
1cbd5fb to
3ea1eb6
Compare
mp_hal_stdio_mode_raw left OPOST enabled, so output newline translation expanded the REPL's \r\n line endings to \r\r\n. This broke the byte-exact mpremote raw REPL protocol over a PTY. Clear OPOST so raw mode produces true raw output. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Extend the event-driven REPL (pyexec_event_repl_*) so it can be driven by an asyncio task when MICROPY_REPL_ASYNCIO is enabled. A complete friendly-REPL line containing "await" is left in MP_STATE_VM(repl_line) and PYEXEC_ASYNC_PENDING is returned for the driver to run on the event loop, after which pyexec_event_repl_resume() re-prompts. Raw mode is signalled via PYEXEC_RAW_ACTIVE so the driver feeds input without yielding. The terminal is switched to original mode around synchronous execution (ISIG for Ctrl-C). pyexec_friendly_repl dispatches to pyexec_asyncio_repl and the blocking body is not compiled when asyncio is enabled. Add pyexec_repl_breakpoint plus readline_push/pop for a reentrant breakpoint REPL. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Expose functions for an asyncio-based REPL, all gated on MICROPY_REPL_ASYNCIO (default off): - stdio_mode_raw(enable): toggle raw terminal mode, behind MICROPY_PY_MICROPYTHON_STDIO_RAW; - repl_event_init / repl_event / repl_event_resume: drive the event-driven REPL a character at a time, returning a status or a deferred await line; - repl(): a blocking breakpoint REPL for use from async code. Document the public functions and add a stdio_mode_raw test. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
asyncio.arepl drives the C event-driven REPL over asyncio: line editing, paste, raw and raw-paste, continuation and synchronous execution all run in C, and only complete lines containing await are executed on the event loop. breakpoint() drops into the blocking repl(). Frozen only when the manifest is included with repl_asyncio=True. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Default MICROPY_REPL_ASYNCIO on at the EXTRA_FEATURES ROM level and freeze asyncio.arepl on the capable ports. Ports that can't run it opt out with MICROPY_REPL_ASYNCIO=0: PYBD_SF2 lacks the flash (also repl_asyncio=False on the stm32 boards manifest), and zephyr/qemu lack a pollable stdin (their consoles use semihosting), so they keep the blocking REPL. rp2 builds its qstr-scan list explicitly, so add shared/runtime/pyexec.c to MICROPY_SOURCE_QSTR there for the event REPL's repl_line root pointer to be collected. Also mark repl_inspect.py flaky under QEMU (REPL timing, like repl_lock) and tidy repl_breakpoint.py: drop unused sys/os imports so it reaches its skip-guard on minimal builds, and a dead readline push/pop branch testing primitives that aren't exposed. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
mp_hal_stdio_poll did not report the stdin ringbuf as readable, so a poll-driven REPL (asyncio reading sys.stdin) never woke on UART, USB CDC or dupterm-notify input. Report MP_STREAM_POLL_RD when the ringbuf has data. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
arepl.task() gains two keyword options for running the REPL as one task among others rather than as the sole boot REPL: - stop_loop_on_exit (default True): stop the event loop on Ctrl-D so the boot REPL soft-resets; False just ends the task and leaves the loop and other tasks running. - persistent (default False): if True, Ctrl-D re-prompts instead of exiting, keeping the console up for the life of the program. The event-driven REPL's Ctrl-D handler now resets the line buffer rather than freeing it, so a persistent caller can re-prompt on the same line. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
A poll-driven reader of sys.stdin (such as asyncio.arepl) registers stdin with the event loop and reads only when poll reports it readable. A dupterm socket the peer closes reports POLL_RD (a read would return EOF), so the reader wakes and calls the blocking stdin read, which consumes the EOF, deactivates the slot, and then has no byte to return, parking the whole event loop until the next byte arrives on any stdin source. On a network-only console that byte never comes, so the REPL wedges until reset. mp_os_dupterm_poll now drops the read-side flags from any slot reporting HUP, ERR or NVAL, i.e. a closed or errored slot, so it no longer makes the aggregated stdin look readable and the reader is not woken into the stranding read. This covers a slot left pointing at an already-closed socket (NVAL), not only one freshly hung up (HUP). modlwip additionally flags a peer-closed TCP socket with no buffered data as HUP (keeping POLL_RD so direct readers still read the EOF), so a clean FIN is masked the same way as a reset. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Under MICROPY_REPL_ASYNCIO the event-driven raw REPL core already implements the same protocol (raw-paste, OK/CTRL-B/CTRL-D framing) as the blocking pyexec_raw_repl() used after a soft reset. Replace its body with a feed loop over the event-driven core so the protocol isn't implemented twice on ports that carry the asyncio REPL. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au> Signed-off-by: Andrew Leech <andrew@alelec.net>
asyncio.arepl.breakpoint() is a nested blocking-REPL debugger convenience built on top of the asyncio REPL, not required for the REPL itself. Move its C support (pyexec_repl_breakpoint(), the readline push/pop reentrant save state, and micropython.repl()) behind a new MICROPY_REPL_ASYNCIO_BREAKPOINT flag, defaulting off below MICROPY_CONFIG_ROM_LEVEL_EVERYTHING so boards that want the asyncio REPL don't pay flash for the debugger by default. The unix "standard" variant enables it explicitly to keep test coverage of tests/micropython/repl_breakpoint.py. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au> Signed-off-by: Andrew Leech <andrew@alelec.net>
andrewleech
force-pushed
the
native_async_repl
branch
from
August 7, 2026 00:39
3ea1eb6 to
eba1b1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds
asyncio.arepl, an interactive REPL that runs as an asyncio task instead of blocking the event loop, so other tasks (Wi-Fi, USB host/device, a socket server, etc) keep running while it waits for and executes input.Rather than reimplementing line editing in Python, it drives the existing event-driven C REPL for paste, raw, raw-paste and continuation handling, and only defers a completed line to the event loop when it contains a top-level
await. That relies on the compiler's existingMICROPY_COMP_ALLOW_TOP_LEVEL_AWAITsupport (already used by the webassembly port), which wraps such a line in a generator so calling it returns a coroutine - no VM changes needed to make a REPL line awaitable. The blocking raw REPL (used after a soft reset) also now feeds through this same event-driven core instead of carrying a second copy of the raw-REPL protocol.task()takesstop_loop_on_exitandpersistentoptions, so it can be the sole boot REPL (Ctrl-D soft-resets) or one task among several that stays up for the life of the program. Ctrl-C at the prompt cancels an in-flightawaitrather than just interrupting the next line. A separateasyncio.arepl.breakpoint()drops into a blocking nested REPL from async code for debugging; its C support is behind its ownMICROPY_REPL_ASYNCIO_BREAKPOINTflag (off by default below theEVERYTHINGROM level) since it's a debugging convenience rather than part of the core REPL.It's on by default at the EXTRA_FEATURES ROM level. A few boards explicitly disable it rather than drop ROM level: PYBD_SF2/SF3 and the esp8266 1M/512K variants are too flash-tight, and zephyr/qemu's stdin isn't pollable. PYBD_SF6 picks it up newly since it previously inherited SF2's disable through shared base config despite having 2 MiB of flash to spare.
Testing
Tested on unix: the cmdline REPL suite (including a new
repl_asyncio_intr.pycovering Ctrl-C cancellation of an in-flightawait) passes, plus the wider test suite and pre-commit's lint/format/spelling checks. Build-verified for stm32 PYBV10 and PYBD_SF6 (flash-size impact below); rp2 RPI_PICO didn't build in this environment due to an unrelated local picotool/CMake version-detection problem, so it's CI-verified there rather than locally rebuilt this round. Earlier hardware testing on an RP2350 (REPL as one task alongside a Wi-Fi supervisor and socket-accept loop, UART and socket REPL both surviving socket churn) predates the later rework that switchedarepl.pyto drive the existing C REPL instead of reimplementing line editing in Python, so that run validates the architecture rather than this exact revision.Trade-offs and Alternatives
Flash cost is about 1.6-1.9 KB on stm32: PYBV10 goes from 374,308 to 376,212 B text+data (+0.51%), PYBD_SF6 from 1,094,952 to 1,096,552 B (+0.15%). Most of that is the extra code paths added to the shared event-driven C REPL plus the new module glue; the frozen
arepl.pyitself is small (under 1 KB) since it only drives the existing REPL and dispatches top-level-await lines to the event loop, rather than reimplementing paste/raw-mode/continuation handling in Python. That reuse is the main design trade-off here: it costs a bit more flash than a from-scratch Python REPL would, but avoids a second implementation of line-editing/paste/raw-mode that would need to stay in sync with the C one. Thebreakpoint()debugger REPL is kept out of that default cost by its own flag.Generative AI
I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the code and the description above.