feat(logloader): MAVLink FTP transport, request-driven fetching, and an HTTP API

[dakejahl]

Summary

Supersedes #29, #30 and #31 — this branch contains their commits unchanged, with authorship intact, so merging this merges all three. They can be closed.

Reviewing that stack turned up structural problems in the code underneath it, and fixing those meant rewriting the parts they sit on, so keeping four PRs would have meant reviewing #29's ServerInterface changes and then reviewing their replacement. One branch instead: MAVLink FTP for transport, fetching driven by request rather than by mirroring the SD card, and an HTTP API for the ARK-OS Logs page to drive.

Verified end to end against PX4 SITL and on hardware: a Jetson upgrading a real pre-FTP install against an ARK FMU v6X on PX4 1.18, exercising the legacy import, the logging-stop and reconnect triggers, armed gating, upload-server outages, and the Logs page through the ARK-OS gateway. Adds tests and CI, which the repo had none of.

Problem

Transport. LOG_DATA (msg 120) has no target_system/target_component, so a router between logloader and the autopilot has no addressing to work with and copies every chunk to every endpoint it serves, telemetry radio included. LOG_ENTRY compounds it: it identifies a log by index and timestamp rather than by file, so anything downloading over FTP while listing over the log protocol has to map one onto the other by size, mtime or list ordering — and ArduPilot's numbering, its LOG_MAX_FILES wrap and its .BIN naming all have to be modelled to do it. When that mapping guesses wrong, the wrong file transfers in full and is discarded.

Structure. ServerInterface was 860 lines doing two unrelated jobs: a SQLite repository and a Flight Review HTTP client. About 70% of it was database code and the name described the other 30%. Because the database lived inside the upload target, download state was duplicated across local_server.db and remote_server.db, and LogLoader kept them in step by calling every mutation twice. The download queue was read from whichever happened to be first.

Behaviour. logloader tried to mirror the vehicle's SD card. A companion powered on for the first time, or one whose database had been reset, would pull and upload the entire history over a link with better things to do — and there was no way to see what it had or to ask for a particular log. It also polled, listing the card on a timer whether or not anything could have changed, and retried failed uploads in a tight loop: on the bench rig journald had suppressed 93,373 messages from the old daemon hammering an offline flight-review.

Defects. A pending log whose local file was missing or empty was retried forever and blocked every older log behind it; only HTTP 400 escaped the queue. ServerInterface::_should_exit and LogLoader::_loop_disabled were plain bools written by one thread and read by another. The signal handler took a mutex and signalled a condition variable, neither of which is async-signal-safe. CMAKE_BUILD_TYPE was never set, so the shipped binary was built at -O0. generate_uuid hashed with std::hash — implementation-defined — into state persisted on disk.

Solution

Everything over MAVLink FTP (FILE_TRANSFER_PROTOCOL, msg 110), for the listing and the transfer. It is addressed, and both stacks reply to the requesting sysid/compid, so transfers are unicast. Identity becomes path-below-the-log-root plus size, so there is no cross-protocol mapping left to get wrong and ArduPilot needs no special handling — a directory listing just reports files. The root is probed at @MAV_LOG, then /fs/microsd/log, /APM/LOGS and /log. MAVSDK's Ftp plugin discards size and mtime from list replies, so FtpListClient runs the listing over MavlinkPassthrough; worth upstreaming so it can be deleted.

Storage split from transport. LogDatabase owns the inventory and the intent, UploadTarget is one Flight Review endpoint with no state of its own, and LegacyImport quarantines the pre-FTP migration behind a header saying it is deletable once the fleet has upgraded. One database, one place to look.

Fetching is request-driven. A log is fetched because it appeared while logloader was watching — a flight just ended — or because something asked. On a database that has never seen the vehicle, only the newest is taken. A listing with more than download.max_auto_queue new logs is an unfamiliar card rather than a burst of flights, so it queues one and says so. Intent lives in the database, so automatic and requested work share one path.

Indexing is event-driven (closes #24). The vehicle is never polled: it is listed when the autopilot connection comes up or comes back, when a flight or logging session ends, and when the API asks (POST /refresh). A flight ending is read from MAV_SYS_STATUS_LOGGING in SYS_STATUS, which PX4 v1.16 and newer raise only while the logger is actually writing — catching logging that runs past disarm or never involved arming (SDLOG_MODE). Older firmware falls back to the disarm transition; ArduPilot raises the bit whenever logging is merely configured, so there it is ignored and disarm is the trigger. Each trigger buys a short burst of listings a few seconds apart — two must agree on a log's size before it counts — and then the loop sleeps; on the bench, 60 seconds of idle fan-out traffic contained zero FILE_TRANSFER_PROTOCOL messages. Armed still pauses indexing and transfers, with owed passes kept for disarm. Connection loss is now actually tracked — set_connected was write-once — and the reconnect edge resets FTP sessions, since a rebooted PX4 can hold a stale one open.

An HTTP API on 127.0.0.1:3005: GET /logs, /status, /events (SSE), POST /logs/{download,upload,cancel}, DELETE /logs/{id}/file. ARK-Electronics/ARK-OS#109 is the page that drives it.

Uploads stream off disk with a computed Content-Length. httplib's MultipartFormDataItems overload concatenates the whole body into a second string; a 400 MB log needed most of a gigabyte on a Pi, and the resulting OOM kill retried the same log on restart. Measured: 16 MB RSS while uploading a 76 MB log.

Also fixed, most of it found by review or by the SITL and hardware runs: a use-after-free where FtpListClient handed MAVSDK a this-capturing callback (MAVSDK unsubscribes asynchronously, so a reply arriving after shutdown wrote into freed heap); an ODR violation from defining CPPHTTPLIB_OPENSSL_SUPPORT per-file, which gave two translation units different socket layouts and segfaulted on the first real run; Transaction::commit clearing its flag before COMMIT, so a failed commit left the transaction open and every later write was silently discarded while the daemon looked healthy; a missing ALTER TABLE for a column added to an existing schema; a log still being written being treated as absent, which marked every row "not on vehicle" for one interval after each restart; a change-detection inversion in sync_index (present rows counted after the wipe, so every identical listing pushed the full log list to every open stream while an emptied card notified nobody); and upload target urls with a trailing slash — which the old flat config shipped — failing httplib's scheme://host:port parse, so a healthy server was reported unreachable forever. That last one was found upgrading the bench rig's real config.

Rows are marked absent rather than deleted when a log leaves the vehicle, so a downloaded log keeps its file and its upload history. Config tables are one level deep on purpose — ARK-OS's shared config editor renders exactly one level, and a setting the operator cannot reach from the web UI may as well not exist. The old flat keys are still read, with a deprecation warning.

Upgrades. Pre-FTP databases keyed logs on the LOG_ENTRY timestamp, which FTP cannot reproduce. local_server.db and remote_server.db are folded into one logloader.db and matched to the listing by size, so a fleet upgrading does not re-download and re-upload its history. A log the old version had downloaded but not yet uploaded keeps its file and its state but is not queued — carrying that intent over would produce exactly the flood this version exists to avoid.

tests/test_log_database.cpp covers 17 cases including the legacy import, which runs once against real user data and cannot be un-run, and the revision semantics the SSE stream depends on. Writing them found two defects on its own.

Carried over from #30 and #31

The reachability cooldown and the API-key headers are preserved in UploadTarget, with one change: 401/403 no longer blacklist a log. Those statuses mean this account is not authorized yet, not this log is bad, and the expected first-run state returns 403 — blacklisting would silently discard one pending log per cycle. The batch still stops on them, which resumes on its own once the key is valid.

Worth flagging: review.arkelectron.com now answers uploads with 403 "requires a registered, approved account (login session or API key)", so the server side has grown key support since this was first written. A key set in upload_remote.api_key is sent as both Authorization: Bearer and X-API-Key; whether those header names match the server's expectation is untested — no approved key was available on the bench. The 403 path itself behaved as designed there: one warning, the batch paused, and the log stayed queued to resume once a key is configured.

Follow-ups

• Bump the ARK-OS submodule pointer — feat(logs-page): add a Logs page for browsing and fetching vehicle logs ARK-OS#109 does this and depends on it.
• Confirm ARK's shipped PX4 configs start the companion MAVLink instance with -x; FTP is now load-bearing.
• Resume interrupted transfers rather than restarting them. A 76 MB log timed out once mid-test and was refetched from scratch; the failure-count ordering keeps it from blocking the queue, but resume is still worth doing.