Add support for running process asynchronously by JoshuaWatt · Pull Request #835 · labgrid-project/labgrid

JoshuaWatt · 2021-09-16T21:22:30Z

Description
Adds support for running processes asynchronously with the main test code. Attempts to incorporate the feed back from #552

Checklist

Documentation for the feature
Tests for the feature

The arguments and description in doc/configuration.rst have been updated

Add a section on how to use the feature to doc/usage.rst

Add a section on how to use the feature to doc/development.rst

CHANGES.rst has been updated

PR has been tested

Man pages have been regenerated

codecov · 2021-09-16T21:28:14Z

Codecov Report

❌ Patch coverage is 94.62366% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.2%. Comparing base (7121ff3) to head (7e729aa).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
labgrid/driver/sshdriver.py	91.6%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #835     +/-   ##
========================================
+ Coverage    61.0%   61.2%   +0.2%     
========================================
  Files         182     183      +1     
  Lines       14906   14999     +93     
========================================
+ Hits         9105    9192     +87     
- Misses       5801    5807      +6

Flag	Coverage Δ
3.10	`61.2% <94.6%> (+0.2%)`	⬆️
3.11	`61.2% <94.6%> (+0.2%)`	⬆️
3.12	`61.2% <94.6%> (+0.1%)`	⬆️
3.13	`61.2% <94.6%> (+0.2%)`	⬆️
3.14	`61.2% <94.6%> (+0.2%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Emantor · 2021-09-17T07:44:37Z

Overall this design looks like what we had in mind in #552. Thank you very much for the implementation, looking forward to testing this on my lab at home.

JoshuaWatt · 2021-09-17T12:51:28Z

It is unfortunately a little hard to write automated tests for most of this since it requires actual hardware interaction, but I'll take a stab at it. I was able to test on actual hardware (Raspberry Pi 4 w/ u-boot) with the following test suite:

import socket
import os
import time

def test_uboot_async(uboot_command):
    with uboot_command.popen("echo hello") as p:
        assert p.read(4).decode("utf-8").splitlines() == ["hell"]
        assert p.read().decode("utf-8").splitlines() == ["o"]

    with uboot_command.popen("echo bzap hello bzap") as p:
        index, before, match, after = p.expect("hello")
        assert index == 0
        assert match.group(0).decode("utf-8") == "hello"
        assert before.decode("utf-8") == "bzap "
        assert after.decode("utf-8") == "hello"
        assert p.read().decode("utf-8").splitlines() == [" bzap"]

def test_ssh_async(ssh_command):
    with ssh_command.popen("echo hello") as p:
        assert p.read(4).decode("utf-8").splitlines() == ["hell"]
        assert p.read().decode("utf-8").splitlines() == ["o"]

    with ssh_command.popen("echo bzap hello bzap") as p:
        index, before, match, after = p.expect("hello")
        assert index == 0
        assert match.group(0).decode("utf-8") == "hello"
        assert before.decode("utf-8") == "bzap "
        assert after.decode("utf-8") == "hello"
        assert p.read().decode("utf-8").splitlines() == [" bzap"]

def test_shell_nc(shell_command, ssh_command):
    with shell_command.popen("nc -l -p 5555") as p:
        # Wait for netcat to start listening
        time.sleep(2)
        with ssh_command.forward_local_port(5555) as local_port:
            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
                s.connect(("localhost", local_port))
                s.sendall(b"Hello World")
            p.expect(b"Hello World")
            time.sleep(2)

def test_ssh_nc(ssh_command):
    with ssh_command.popen("nc -l -p 5555 127.0.0.1") as p:
        # Wait for netcat to start listening
        time.sleep(2)
        with ssh_command.forward_local_port(5555) as local_port:
            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
                s.connect(("localhost", local_port))
                s.sendall(b"Hello World")
            p.expect(b"Hello World")
            time.sleep(2)


def test_shell_async(shell_command):
    with shell_command.popen("echo hello") as p:
        assert p.read(4).decode("utf-8").splitlines() == ["hell"]
        assert p.read().decode("utf-8").splitlines() == ["o"]

    with shell_command.popen("echo bzap hello bzap") as p:
        index, before, match, after = p.expect("hello")
        assert index == 0
        assert match.group(0).decode("utf-8") == "hello"
        assert before.decode("utf-8") == "bzap "
        assert after.decode("utf-8") == "hello"
        assert p.read().decode("utf-8").splitlines() == [" bzap"]

def test_shell_interrupt(shell_command):
    with shell_command.popen("yes") as p:
        l = p.read(1024).decode("utf-8").splitlines()
        assert l == ["y"] * len(l)

    shell_command.run_check("true")

Note that I wasn't able to test the barebox driver, so manual testing there would be appreciated

JoshuaWatt · 2021-09-17T19:39:07Z

I added some tests, which revealed a few bugs that needed fixed. I think it's all good to go now though (although I still have no way to test barebox)

Bastian-Krause

Great work! I found some trivial things. I'll have to dive deeper into this later.

This should be mentioned in CHANGES.rst and probably needs some documentation elsewhere?

JoshuaWatt · 2021-09-20T13:20:26Z

Great work! I found some trivial things. I'll have to dive deeper into this later.

This should be mentioned in CHANGES.rst and probably needs some documentation elsewhere?

I added a little to CHANGES.rst; I can add more documentation but I'm not really sure where it should go.

Bastian-Krause

I've tested this briefly now.

Issues I stumbled upon:

BareboxDriver now times out on run()
running multiple processes on BareboxDriver/ShellDriver is not forbidden, but runs in timeouts
calling stop() on ConsoleMarkerProcess returned by ShellDriver.popen() leads to a timeout (e.g. stopping a "sleep 100") - the implementation looks incomplete, I don't see the prompt being detected on stop()

Open questions:

why is SSHDriver.run() not using popen()?
I do not really understand the difference between read_nonblocking()/read()

Styling:

we use f-strings where appropriate since #838, please do the same

Note: this is no comprehensive review yet, just a collection of things I saw while testing with a QEMU and a physical target.

Bastian-Krause · 2021-09-28T18:25:15Z

+
+    @step(args=['size', 'timeout'], result=True)
+    def read(self, size=-1, *, timeout=None):
+        t = OptionalTimeout(timeout, self._console.timeout)


QEMUDriver and ExternalConsoleDriver do not implement a timeout attribute.

Bastian-Krause · 2021-09-28T18:32:59Z

+                output = p.read(timeout=timeout)
+                # Remove VT100 Codes and split by newline
+                data = self.re_vt100.sub('', output.decode('utf-8')).split('\r\n')[1:-1]
+                self.logger.debug(f"Received Data: {data}")


log functions should not use f-strings, but rather lazy % formatting. There are multiple occurrences of this spread in this PR.

Is there a reason why? It seem confusing to developers to mix the two styles without a good reason :/

Yes, there is a reason. Here is a short wrap-up: https://medium.com/swlh/why-it-matters-how-you-log-in-python-1a1085851205

TL;DR: formatting is only done if required (i.e. the log level is reached) and errors are handled gracefully.

Reading the article has confused me more; IIUC, the article is suggesting that logging should use lazy evaluation by passing format arguments as parameters (not using %), in which case the log message would be something like self.logger.debug("Received data: %s", data). However, it seems you are suggesting I use % instead, which is not a lazy evalutation and therefore not superior (at least in terms of what the article is discussing) from f-strings.

I think what you want to do is either use actual lazy evaluation for log messages, or non-lazy f-strings for consistency with everywhere else f-strings are used

With "% formatting" I mean logger.debug("%s", data).

OK, I will do that

Bastian-Krause · 2021-09-28T18:35:35Z

 - AndroidFastbootDriver now supports booting/flashing images preconfigured in
  the environment configuration.
+- CommandProtocol now supports a popen() method to run a process on the target
+  asynchronously with the pytest code.


Why mention pytest here? The PR does not touch anything pytest-specific.

Bastian-Krause · 2021-09-28T18:37:05Z

+                data = self.re_vt100.sub(
+                    '', output.decode('utf-8'), count=1000000
+                ).replace("\r", "").split("\n")
+                self.logger.debug("Received Data: {data}")


I guess this is meant as an f-string. Please convert to lazy % formatting.

Per previous discussion, % is not lazy

Bastian-Krause · 2021-09-28T18:39:36Z


 def gen_marker():
-    return ''.join(random.choice(string.ascii_uppercase) for i in range(10))
+    return "".join(random.choice(string.ascii_uppercase) for i in range(10))


No need to change this line.

Bastian-Krause · 2021-09-28T18:43:26Z

+    will never expire. If timeout negative and default is default is not None,
+    the default timer value will be used


This sentence needs revisiting.

JoshuaWatt · 2021-09-28T20:20:35Z

I've tested this briefly now.

Issues I stumbled upon:
* `BareboxDriver` now times out on `run()`

I don't have any way of testing barebox, can you provide more information so I can fix it?

* running multiple processes on `BareboxDriver`/`ShellDriver` is not forbidden, but runs in timeouts

Do we want to explicitly disallow that? I'm fine with that if that's what everyone wants to do.

* calling `stop()` on `ConsoleMarkerProcess` returned by `ShellDriver.popen()` leads to a timeout (e.g. stopping a "sleep 100") - the implementation looks incomplete, I don't see the prompt being detected on `stop()`

Open questions:

* why is `SSHDriver.run()` not using `popen()`?

It would be nice, but I can't do that without breaking the SSH driver because of the stderr_merge option, which is not possible to implement with popen()

* I do not really understand the difference between `read_nonblocking()`/`read()`

read_nonblocking allows a CommandProcessProcotol to be directly used as the backend for a popen.spawn object, since the latter requires the the read_nonblocking function to be implemented. read is the API for normal folks :)

Styling:

* we use f-strings where appropriate since #838, please do the same

I tried to use f-strings everywhere.... IHMO it's confusing to developers to require different string formatting for different uses (unless there is some really compelling performance reason to do otherwise).

Note: this is no comprehensive review yet, just a collection of things I saw while testing with a QEMU and a physical target.

Bastian-Krause · 2021-09-28T21:49:44Z

BareboxDriver now times out on run()

I don't have any way of testing barebox, can you provide more information so I can fix it?

Here is my demo script:

#!/usr/bin/env python3

import labgrid

labgrid.StepReporter()

env = labgrid.Environment("beaglebone.yaml")
target = env.get_target()

barebox = target.get_driver("BareboxDriver")
target.activate(barebox)

barebox.run("echo hi")

This results in:
labgrid-barebox-run.log

I have not looked into it yet.

JoshuaWatt · 2021-10-18T14:09:03Z

I think I've addressed most of the comment, is there anything else needed to merge this?

Bastian-Krause · 2021-10-19T08:52:07Z

@JoshuaWatt Thanks, re-requesting review or a short notice like yours is always welcome, so we know when to look at PRs again. I need to look at/test the current state again. I'll come back to you.

Bastian-Krause · 2021-10-28T10:28:52Z

How would you run a command that takes longer than 30 seconds?

def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass

This ends up with a TIMEOUT exception.

Another thing: how would you read the output of a long-running command (e.g. find /) asynchronously? Could you provide an example? I tried something with while p.poll() is None, but the poll() method blocks for a second if the process is still alive, so that feels wrong.

Bastian-Krause · 2021-10-28T11:29:40Z

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
This ends up with a TIMEOUT exception.

Ah, this has nothing to do with the long-running command. The context manager hits CTRL-c on close, which prevents the second marker from being echoed (on the shell, not in barebox though). That's why we run into the timeout there.

JoshuaWatt · 2021-11-02T17:58:15Z

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
This ends up with a TIMEOUT exception.

Another thing: how would you read the output of a long-running command (e.g. find /) asynchronously? Could you provide an example? I tried something with while p.poll() is None, but the poll() method blocks for a second if the process is still alive, so that feels wrong.

poll is for checking if the process is still alive; to read output use p.read() or p.expect()

JoshuaWatt · 2021-11-02T18:00:27Z

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
This ends up with a TIMEOUT exception.
Ah, this has nothing to do with the long-running command. The context manager hits CTRL-c on close, which prevents the second marker from being echoed (on the shell, not in barebox though). That's why we run into the timeout there.

Does something need to change? It seems like it's doing what you are asking for (launching the process, then immediately terminating it). If you want to wait for it to exit normally, use p.poll() in a loop, or p.wait()

Bastian-Krause · 2021-11-03T11:11:42Z

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
This ends up with a TIMEOUT exception.
Another thing: how would you read the output of a long-running command (e.g. find /) asynchronously? Could you provide an example? I tried something with while p.poll() is None, but the poll() method blocks for a second if the process is still alive, so that feels wrong.
poll is for checking if the process is still alive; to read output use p.read() or p.expect()

Okay, will try that.

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
  
This ends up with a `TIMEOUT` exception.
Ah, this has nothing to do with the long-running command. The context manager hits CTRL-c on close, which prevents the second marker from being echoed (on the shell, not in barebox though). That's why we run into the timeout there.
Does something need to change? It seems like it's doing what you are asking for (launching the process, then immediately terminating it). If you want to wait for it to exit normally, use p.poll() in a loop, or p.wait()

I think the behavior between barebox and shell should be consistent. I would not expect to get a TIMEOUT exception if I do not wait for the process to finish. Do you agree?

JoshuaWatt · 2021-11-03T13:00:28Z

How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
This ends up with a TIMEOUT exception.
Another thing: how would you read the output of a long-running command (e.g. find /) asynchronously? Could you provide an example? I tried something with while p.poll() is None, but the poll() method blocks for a second if the process is still alive, so that feels wrong.
poll is for checking if the process is still alive; to read output use p.read() or p.expect()
Okay, will try that.
How would you run a command that takes longer than 30 seconds?
def test_async_output(strategy):
    strategy.transition("shell")
    with strategy.shell.popen("sleep 40") as p:
        pass
  
This ends up with a `TIMEOUT` exception.
Ah, this has nothing to do with the long-running command. The context manager hits CTRL-c on close, which prevents the second marker from being echoed (on the shell, not in barebox though). That's why we run into the timeout there.
Does something need to change? It seems like it's doing what you are asking for (launching the process, then immediately terminating it). If you want to wait for it to exit normally, use p.poll() in a loop, or p.wait()
I think the behavior between barebox and shell should be consistent. I would not expect to get a TIMEOUT exception if I do not wait for the process to finish. Do you agree?

Ya, that makes sense; let me see what I can do

JoshuaWatt · 2021-11-03T14:26:35Z

@Bastian-Krause Ok, I tracked this down to a behavior difference is bash (which I was using) vs. busybox. I've corrected it and now it should work the same for both shells

JoshuaWatt · 2021-11-03T14:30:10Z

For those interested, the difference is:

bash:

raspberrypi4-64:~# echo -n 'ABC'; sleep 10; echo 'ABC' $?; 
ABC^C
ABC 130
raspberrypi4-64:~#

busybox:

raspberrypi4-64:~# echo -n 'ABC'; sleep 10; echo 'ABC' $?;
ABC^C
raspberrypi4-64:~#

Note that interrupting the command on busybox prevents all subsequent commands (the echo with the exit code) from running, where as they still run under bash.

JPEWdev · 2025-08-14T20:11:56Z

@Bastian-Krause I think this is ready, would you mind looking again?

Bastian-Krause · 2025-10-01T09:57:24Z

Thanks for keeping at it! I'll try to have a look at this and run our network test suite against this.

Bastian-Krause · 2026-02-02T10:06:02Z

Thanks for keeping at it! I'll try to have a look at this and run our network test suite against this.

Due to our tasks shifting, I did not work on our (currently internal) test suite, thus did not manage to get this tested. Sorry. If anyone else could give this a spin, that would help.

jluebbe · 2026-02-18T16:31:52Z

+    """labgrid Wrapper of the pexpect module.
+
+    This class provides pexpect functionality for the ConsoleProtocol classes.
+    driver: ConsoleProtocol object to be passed in


This should explain why we need this in addition to the PtxExpect wrapper in labgrid/util/expect.py. It seems this allows the user to use expect only on the process output, without having to care about the markers.

jluebbe · 2026-02-18T16:32:25Z

+    def send(self, s):
+        raise NotImplementedError("Sending not implemented")


Is there a reason why we don't want to implement send?

JoshuaWatt · 2026-06-10T16:36:47Z

Would someone mind looking at this change again? Thanks

Emantor

I gave this a once over and everything seems to be addressed. We just had a release, so its a good time to get this in for additional testing.

Emantor · 2026-07-08T08:08:39Z

We took a deeper look with @jluebbe, @Bastian-Krause and me and taking the state of our serial stack into account, we are not really comfortable to add so much additional complexity on top of the rather fragile expect matching. Nevertheless, the changes for the SSHDriver are much less invasive and more straightforward. What do you think about splitting these out and getting them merged?

While reviewing we also wondered whether the background functionality for the bootloader drivers is really necessary, do you have examples for usage or were those implemented for CommandProtocol completeness?

JoshuaWatt · 2026-07-08T15:07:36Z

We took a deeper look with @jluebbe, @Bastian-Krause and me and taking the state of our serial stack into account, we are not really comfortable to add so much additional complexity on top of the rather fragile expect matching. Nevertheless, the changes for the SSHDriver are much less invasive and more straightforward. What do you think about splitting these out and getting them merged?

While reviewing we also wondered whether the background functionality for the bootloader drivers is really necessary, do you have examples for usage or were those implemented for CommandProtocol completeness?

Yes, it was mostly done for completeness. I don't beleive we would need serial async execution. Do you want the serial protocols to raise a NotImplementedError for CommandProtocol.start_process() or would you prefer start_process() be part of a new protocol?

Emantor · 2026-07-08T15:52:58Z

We should just implement it on top of the SSHDriver and not use a protocol at all. If we ever find a second user we can still extend/create a protocol.

Allowing the class to take an integer for a timeout instead of a strict float seems more convenient to users than forcing it at a higher level, or breaking usage by changing other APIs to require floats (i.e. run_check()) Signed-off-by: Joshua Watt <Joshua.Watt@garmin.com>

Adds a read mix in class that implements read helper functions to deal with reading from sources that will return only buffered data, instead of trying harder to read up to the user requested size. These functions are modeled after the functions provided by Rust Signed-off-by: Joshua Watt <Joshua.Watt@garmin.com>

Implements Asynchronous API for SSH connections. The SSHDriverProcess is a wrapper around pexpect.spawn() Change-Id: Icf5fabe7d0b21a4c65bc9b9cc8c5d9849f2faae1 Signed-off-by: Joshua Watt <Joshua.Watt@garmin.com>

Bastian-Krause

Since SSHDriver.start_process() merges stdout and stderr, I'm wondering if it should raise NotImplementedError if its stderr_merge attribute is False?

JoshHollis251 · 2026-08-18T16:38:21Z

Since SSHDriver.start_process() merges stdout and stderr, I'm wondering if it should raise NotImplementedError if its stderr_merge attribute is False?

If we require stderr_merge=false and the ssh_driver instance were to perform a .run() following a start_process() operation it would have the side effect of maybe unintentionally merging those fields unless it is set back to the default. Is that acceptable behavior, and should we include a unit test for the not_implemented check?

JoshuaWatt · 2026-08-18T16:39:52Z

FYI: @JoshHollis251 is helping me finish this PR :)

Emantor added the enhancement label Sep 17, 2021

JoshuaWatt force-pushed the command-background branch from a336b91 to 1013b98 Compare September 17, 2021 12:44

JoshuaWatt force-pushed the command-background branch from 1013b98 to 6dd0e24 Compare September 17, 2021 19:16

Bastian-Krause previously requested changes Sep 20, 2021

View reviewed changes

JoshuaWatt force-pushed the command-background branch from 6dd0e24 to f837162 Compare September 20, 2021 13:18

Bastian-Krause reviewed Sep 20, 2021

View reviewed changes

Comment thread labgrid/protocol/commandprotocol.py Outdated

JoshuaWatt force-pushed the command-background branch 2 times, most recently from 580e931 to 88b4826 Compare September 22, 2021 14:07

Emantor added this to the v0.5.0 milestone Sep 22, 2021

Bastian-Krause reviewed Sep 28, 2021

View reviewed changes

JoshuaWatt force-pushed the command-background branch 2 times, most recently from a2719bd to 53b2f8c Compare October 4, 2021 15:59

JoshuaWatt force-pushed the command-background branch from 53b2f8c to 83c64e9 Compare November 3, 2021 14:25

adrian-martin mentioned this pull request May 19, 2025

SSHDriver fails to launch commands in the background #1658

Open

JoshuaWatt force-pushed the command-background branch from 58d847d to ba12014 Compare June 3, 2025 19:11

JoshuaWatt force-pushed the command-background branch 2 times, most recently from a9e0e17 to 8546a36 Compare June 12, 2025 20:56

JoshuaWatt force-pushed the command-background branch 2 times, most recently from 489db0e to ac03245 Compare July 3, 2025 19:53

Emantor force-pushed the command-background branch from abe5a95 to f6b278f Compare February 18, 2026 15:40

Emantor added this to the v26.1 milestone Feb 18, 2026

jluebbe reviewed Feb 18, 2026

View reviewed changes

JoshuaWatt force-pushed the command-background branch 3 times, most recently from 8e510d5 to 84402d4 Compare February 23, 2026 23:23

Emantor previously approved these changes Jun 11, 2026

View reviewed changes

Bastian-Krause added the architectural change label Jul 8, 2026

Bastian-Krause added the needs author info Requires more information from the PR/Issue author label Aug 7, 2026

JoshuaWatt added 3 commits August 12, 2026 14:40

sshdriver: Implement Asynchronous Processes

7e729aa

Implements Asynchronous API for SSH connections. The SSHDriverProcess is a wrapper around pexpect.spawn() Change-Id: Icf5fabe7d0b21a4c65bc9b9cc8c5d9849f2faae1 Signed-off-by: Joshua Watt <Joshua.Watt@garmin.com>

JoshuaWatt dismissed Emantor’s stale review via 7e729aa August 12, 2026 20:40

JoshuaWatt force-pushed the command-background branch from ad293fa to 7e729aa Compare August 12, 2026 20:40

Bastian-Krause reviewed Aug 15, 2026

View reviewed changes

		will never expire. If timeout negative and default is default is not None,
		the default timer value will be used

		def send(self, s):
		raise NotImplementedError("Sending not implemented")

Uh oh!

Uh oh!

Conversation

JoshuaWatt commented Sep 16, 2021

Uh oh!

codecov Bot commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Emantor commented Sep 17, 2021

Uh oh!

JoshuaWatt commented Sep 17, 2021

Uh oh!

JoshuaWatt commented Sep 17, 2021

Uh oh!

Bastian-Krause left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshuaWatt commented Sep 20, 2021

Uh oh!

Uh oh!

Bastian-Krause left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshuaWatt Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshuaWatt Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshuaWatt commented Sep 28, 2021

Uh oh!

Bastian-Krause commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshuaWatt commented Oct 18, 2021

Uh oh!

Bastian-Krause commented Oct 19, 2021

Uh oh!

Bastian-Krause commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bastian-Krause commented Oct 28, 2021

Uh oh!

JoshuaWatt commented Nov 2, 2021

Uh oh!

JoshuaWatt commented Nov 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

codecov Bot commented Sep 16, 2021 •

edited

Loading

Bastian-Krause left a comment •

edited

Loading

JoshuaWatt Sep 28, 2021 •

edited

Loading

JoshuaWatt Oct 1, 2021 •

edited

Loading

Bastian-Krause commented Sep 28, 2021 •

edited

Loading

Bastian-Krause commented Oct 28, 2021 •

edited

Loading

JoshuaWatt commented Nov 2, 2021 •

edited

Loading

JoshuaWatt commented Nov 3, 2021 •

edited

Loading

JoshHollis251 commented Aug 18, 2026 •

edited

Loading