Skip to content

extmod/asyncio: Add a native asyncio REPL - #49

Draft
andrewleech wants to merge 193 commits into
review/native_async_replfrom
native_async_repl
Draft

extmod/asyncio: Add a native asyncio REPL#49
andrewleech wants to merge 193 commits into
review/native_async_replfrom
native_async_repl

Conversation

@andrewleech

@andrewleech andrewleech commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Summary

This adds asyncio.arepl, an interactive REPL that runs as an asyncio task instead of blocking the event loop, so other tasks (Wi-Fi, USB host/device, a socket server, etc) keep running while it waits for and executes input.

Rather than reimplementing line editing in Python, it drives the existing event-driven C REPL for paste, raw, raw-paste and continuation handling, and only defers a completed line to the event loop when it contains a top-level await. That relies on the compiler's existing MICROPY_COMP_ALLOW_TOP_LEVEL_AWAIT support (already used by the webassembly port), which wraps such a line in a generator so calling it returns a coroutine - no VM changes needed to make a REPL line awaitable. The blocking raw REPL (used after a soft reset) also now feeds through this same event-driven core instead of carrying a second copy of the raw-REPL protocol.

task() takes stop_loop_on_exit and persistent options, so it can be the sole boot REPL (Ctrl-D soft-resets) or one task among several that stays up for the life of the program. Ctrl-C at the prompt cancels an in-flight await rather than just interrupting the next line. A separate asyncio.arepl.breakpoint() drops into a blocking nested REPL from async code for debugging; its C support is behind its own MICROPY_REPL_ASYNCIO_BREAKPOINT flag (off by default below the EVERYTHING ROM level) since it's a debugging convenience rather than part of the core REPL.

It's on by default at the EXTRA_FEATURES ROM level. A few boards explicitly disable it rather than drop ROM level: PYBD_SF2/SF3 and the esp8266 1M/512K variants are too flash-tight, and zephyr/qemu's stdin isn't pollable. PYBD_SF6 picks it up newly since it previously inherited SF2's disable through shared base config despite having 2 MiB of flash to spare.

Testing

Tested on unix: the cmdline REPL suite (including a new repl_asyncio_intr.py covering Ctrl-C cancellation of an in-flight await) passes, plus the wider test suite and pre-commit's lint/format/spelling checks. Build-verified for stm32 PYBV10 and PYBD_SF6 (flash-size impact below); rp2 RPI_PICO didn't build in this environment due to an unrelated local picotool/CMake version-detection problem, so it's CI-verified there rather than locally rebuilt this round. Earlier hardware testing on an RP2350 (REPL as one task alongside a Wi-Fi supervisor and socket-accept loop, UART and socket REPL both surviving socket churn) predates the later rework that switched arepl.py to drive the existing C REPL instead of reimplementing line editing in Python, so that run validates the architecture rather than this exact revision.

Trade-offs and Alternatives

Flash cost is about 1.6-1.9 KB on stm32: PYBV10 goes from 374,308 to 376,212 B text+data (+0.51%), PYBD_SF6 from 1,094,952 to 1,096,552 B (+0.15%). Most of that is the extra code paths added to the shared event-driven C REPL plus the new module glue; the frozen arepl.py itself is small (under 1 KB) since it only drives the existing REPL and dispatches top-level-await lines to the event loop, rather than reimplementing paste/raw-mode/continuation handling in Python. That reuse is the main design trade-off here: it costs a bit more flash than a from-scratch Python REPL would, but avoids a second implementation of line-editing/paste/raw-mode that would need to stay in sync with the C one. The breakpoint() debugger REPL is kept out of that default cost by its own flag.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the code and the description above.

@codecov-commenter

codecov-commenter commented Jun 16, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.54%. Comparing base (13303f8) to head (eba1b1f).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@                     Coverage Diff                      @@
##           review/native_async_repl      #49      +/-   ##
============================================================
+ Coverage                     98.50%   98.54%   +0.03%     
============================================================
  Files                           177      182       +5     
  Lines                         22927    23332     +405     
  Branches                          0        5       +5     
============================================================
+ Hits                          22584    22992     +408     
+ Misses                          343      339       -4     
- Partials                          0        1       +1     
Flag Coverage Δ
unix-coverage-32bit 98.54% <100.00%> (?)
unix-coverage-64bit 98.47% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

Code size report:

Reference:  py/py.mk: Move MICROPY_FORCE_32BIT to the Windows port's Makefile. [13303f8]
Comparison: extmod/asyncio: Gate breakpoint() debugger behind its own flag. [merge of eba1b1f]
  mpy-cross:  +560 +0.147% 
   bare-arm:   +28 +0.050% 
minimal x86:   +69 +0.037% 
   unix x64: +6152 +0.713% standard[incl +480(data) +72(bss)]
      stm32: +2940 +0.730% PYBV10[incl +16(bss)]
      esp32: +12848 +0.728% ESP32_GENERIC[incl +2368(data) +48(bss)]
     mimxrt: +2248 +0.571% TEENSY40
        rp2: +12420 +1.307% RPI_PICO_W[incl -540(bss)]
       samd: +2308 +0.831% ADAFRUIT_ITSYBITSY_M4_EXPRESS[incl +44(bss)]
  qemu rv32:  +763 +0.167% VIRT_RV32

@andrewleech
andrewleech force-pushed the native_async_repl branch 3 times, most recently from e66b858 to 39c82bb Compare June 18, 2026 06:43
rafal-kw and others added 20 commits July 6, 2026 22:50
This commit adds support for the ESP32-H2 processor from Espressif in the
form of two board definitions - ESP32_GENERIC_H2 and M5STACK_NANOH2.

Completed testing on the M5StackH2:
- tested GPIO via on board LED
- tested ADC channels can be read
- I2C & SPI instantiation
- BLE packet scanner
- run_tests.py

Signed-off-by: Rafal Wadowski <rafal@wadowski.com.au>
Added pins for I2S.

Added ESP32-H2 check to reduce number of SPI peripherals.

Signed-off-by: Rafal Wadowski <rafal@wadowski.com.au>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Raises LookupError for not implemented error handlers.

Improves repr() rendering for unicode.

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Only accepts `utf-8`, `utf8` or `ascii`

Fixes micropython#15849

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Fixes: issue 3364
Fixes: issue 13084

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Fixes Issue 17827

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Prevent the test from failing by not testing known unsupported characters.
These will be documented in a cpydiff test.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
These are now printed as characters rather than escaped bytes.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
@andrewleech
andrewleech force-pushed the native_async_repl branch 2 times, most recently from 2b262f0 to 1cbd5fb Compare July 8, 2026 01:36
dpgeorge added 3 commits July 8, 2026 23:16
Three tests are rewritten using unittest, so that they don't need to be run
under CPython to get the expected output.  Because running under CPython
requires port 8000 to be available, which it may not (at least for tests in
the extmod directory having a port available should not be a requirement).

Signed-off-by: Damien George <damien@micropython.org>
Similar to the parent commit, this means CPython doesn't need to run to get
the expected output, and means port 8000 doesn't need to be available.

Signed-off-by: Damien George <damien@micropython.org>
MicroPython allows unicode to be disabled to reduce firmware size, in which
case str objects are essentially the same as bytes objects when it comes to
code points between 128 and 255.  But that behaviour has subtle differences
to when unicode is enabled, eg string lengths can be different.

Since unicode strings are arguably an important feature of Python, and
users would likely be tripped up with it disabled, this commit changes the
default level at which unicode is enabled from "extra" down to "basic".

Costs about +1600 bytes on ARM Cortex-M targets.

Signed-off-by: Damien George <damien@micropython.org>
projectgus and others added 18 commits July 30, 2026 16:42
Requires a tweak in makeqstrdefs to not enable debug output from the
preprocessor, as -g3 adds a lot of extra macro definitions - some of which
are picked up by makeqstrdefs otherwise.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
This variant has the smallest application partition in flash.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
The board's on-board SPI flash chip footprint shares pins with the
side gpio headers, but the README only mentioned this in general
terms.  Add an explicit "Pin notes" subsection that lists which pins
are used by SPI1 / FLASH_* per variant, calls out the SPI3 conflict
on v2.0, and documents the empirical PB4 behaviour on v3.1 reported
in issue micropython#18432.

Fixes micropython#18432.

Signed-off-by: Andrii Anoshyn <anoshyn.andrii@gmail.com>
Signed-off-by: Dryw Wade <dryw.wade@sparkfun.com>
Set a flag in the link-up transition and MAC-reconfig branches of
eth_phy_link_status_poll(), then call eth_dhcp_restart_if_needed()
once at the end of the function guarded by the flag.  Avoids the
back-to-back stop/start that occurred when both branches ran in the
same poll iteration.  Reported by @dpgeorge in the PR review of the
original hot-plug support.

Signed-off-by: Andrew Leech <andrew@alelec.net>
The transition block just above unconditionally assigns
self->last_link_status = current_link_status, and the surrounding
if condition already tested current_link_status true.  So by the
time this check runs, last_link_status is always true here and the
early return can never fire.

Signed-off-by: Andrew Leech <andrew@alelec.net>
Fixes micropython#19505. A regression was introduced in e48b985,
where the USB sleep clock got enabled in stm32/main.c
instead of the shared USB code, so mboot/main.c no longer
enabled the USB sleep clock, causing failed enumeration.

Signed-off-by: Dryw Wade <dryw.wade@sparkfun.com>
BUTTON was renamed to SW in 64c7045.

Signed-off-by: Damien George <damien@micropython.org>
This allows devices to support partial update of the ROM segments.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This uses the new GET_MIN_PREPARE ioctl to support incremental preparation
of the ROM partition.  On very large partitions the preparation can take a
long time (eg due to the erase time of many blocks) and this can lead to a
timeout of mpremote.  Using incremental preparation prevents the timeout
and gives better feedback to the user as to the progress.

Signed-off-by: Damien George <damien@micropython.org>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: jaenrig-ifx <enriquezgarcia.external@infineon.com>
Signed-off-by: jaenrig-ifx <enriquezgarcia.external@infineon.com>
external_call_depth is only read back (to detect the top-level call
boundary) when MICROPY_GC_SPLIT_HEAP_AUTO is enabled. The "standard"
variant does not enable this option, so its build was failing with:

    main.c:50:15: error: variable 'external_call_depth' set but not
    used [-Werror,-Wunused-but-set-global]

Fix by guarding the variable's declaration, increment, and decrement
all under the same #if, instead of leaving the decrement outside the
guard while the increment and declaration were inside/outside
inconsistently.

Fixes regression introduced by cdaf2de.

Signed-off-by: Harris <harrisjan06@gmail.com>
pi-anl and others added 10 commits August 7, 2026 10:06
mp_hal_stdio_mode_raw left OPOST enabled, so output newline translation
expanded the REPL's \r\n line endings to \r\r\n.  This broke the
byte-exact mpremote raw REPL protocol over a PTY.  Clear OPOST so raw
mode produces true raw output.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Extend the event-driven REPL (pyexec_event_repl_*) so it can be driven
by an asyncio task when MICROPY_REPL_ASYNCIO is enabled.  A complete
friendly-REPL line containing "await" is left in MP_STATE_VM(repl_line)
and PYEXEC_ASYNC_PENDING is returned for the driver to run on the event
loop, after which pyexec_event_repl_resume() re-prompts.  Raw mode is
signalled via PYEXEC_RAW_ACTIVE so the driver feeds input without
yielding.  The terminal is switched to original mode around synchronous
execution (ISIG for Ctrl-C).  pyexec_friendly_repl dispatches to
pyexec_asyncio_repl and the blocking body is not compiled when asyncio
is enabled.  Add pyexec_repl_breakpoint plus readline_push/pop for a
reentrant breakpoint REPL.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Expose functions for an asyncio-based REPL, all gated on
MICROPY_REPL_ASYNCIO (default off):

- stdio_mode_raw(enable): toggle raw terminal mode, behind
  MICROPY_PY_MICROPYTHON_STDIO_RAW;
- repl_event_init / repl_event / repl_event_resume: drive the
  event-driven REPL a character at a time, returning a status or a
  deferred await line;
- repl(): a blocking breakpoint REPL for use from async code.

Document the public functions and add a stdio_mode_raw test.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
asyncio.arepl drives the C event-driven REPL over asyncio: line editing,
paste, raw and raw-paste, continuation and synchronous execution all run
in C, and only complete lines containing await are executed on the event
loop.  breakpoint() drops into the blocking repl().  Frozen only when the
manifest is included with repl_asyncio=True.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Default MICROPY_REPL_ASYNCIO on at the EXTRA_FEATURES ROM level and
freeze asyncio.arepl on the capable ports.

Ports that can't run it opt out with MICROPY_REPL_ASYNCIO=0: PYBD_SF2
lacks the flash (also repl_asyncio=False on the stm32 boards manifest),
and zephyr/qemu lack a pollable stdin (their consoles use semihosting),
so they keep the blocking REPL.

rp2 builds its qstr-scan list explicitly, so add shared/runtime/pyexec.c
to MICROPY_SOURCE_QSTR there for the event REPL's repl_line root pointer
to be collected.

Also mark repl_inspect.py flaky under QEMU (REPL timing, like repl_lock)
and tidy repl_breakpoint.py: drop unused sys/os imports so it reaches
its skip-guard on minimal builds, and a dead readline push/pop branch
testing primitives that aren't exposed.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
mp_hal_stdio_poll did not report the stdin ringbuf as readable, so a
poll-driven REPL (asyncio reading sys.stdin) never woke on UART, USB CDC
or dupterm-notify input.  Report MP_STREAM_POLL_RD when the ringbuf has
data.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
arepl.task() gains two keyword options for running the REPL as one task
among others rather than as the sole boot REPL:

- stop_loop_on_exit (default True): stop the event loop on Ctrl-D so the
  boot REPL soft-resets; False just ends the task and leaves the loop and
  other tasks running.
- persistent (default False): if True, Ctrl-D re-prompts instead of
  exiting, keeping the console up for the life of the program.

The event-driven REPL's Ctrl-D handler now resets the line buffer rather
than freeing it, so a persistent caller can re-prompt on the same line.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
A poll-driven reader of sys.stdin (such as asyncio.arepl) registers
stdin with the event loop and reads only when poll reports it readable.
A dupterm socket the peer closes reports POLL_RD (a read would return
EOF), so the reader wakes and calls the blocking stdin read, which
consumes the EOF, deactivates the slot, and then has no byte to return,
parking the whole event loop until the next byte arrives on any stdin
source.  On a network-only console that byte never comes, so the REPL
wedges until reset.

mp_os_dupterm_poll now drops the read-side flags from any slot reporting
HUP, ERR or NVAL, i.e. a closed or errored slot, so it no longer makes
the aggregated stdin look readable and the reader is not woken into the
stranding read.  This covers a slot left pointing at an already-closed
socket (NVAL), not only one freshly hung up (HUP).  modlwip additionally
flags a peer-closed TCP socket with no buffered data as HUP (keeping
POLL_RD so direct readers still read the EOF), so a clean FIN is masked
the same way as a reset.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Under MICROPY_REPL_ASYNCIO the event-driven raw REPL core already
implements the same protocol (raw-paste, OK/CTRL-B/CTRL-D framing) as
the blocking pyexec_raw_repl() used after a soft reset. Replace its
body with a feed loop over the event-driven core so the protocol
isn't implemented twice on ports that carry the asyncio REPL.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew@alelec.net>
asyncio.arepl.breakpoint() is a nested blocking-REPL debugger convenience
built on top of the asyncio REPL, not required for the REPL itself. Move
its C support (pyexec_repl_breakpoint(), the readline push/pop reentrant
save state, and micropython.repl()) behind a new
MICROPY_REPL_ASYNCIO_BREAKPOINT flag, defaulting off below
MICROPY_CONFIG_ROM_LEVEL_EVERYTHING so boards that want the asyncio REPL
don't pay flash for the debugger by default.

The unix "standard" variant enables it explicitly to keep test coverage
of tests/micropython/repl_breakpoint.py.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew@alelec.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.