Remove "hybrid" dev approach, fix tests and local dev scripting

[jechols]

The "hybrid" dev approach works great... in docker-compose. And as long as you know a few key caveats / oddities / etc.

There are quite a few problems I'm noticing, having switched to podman. First, some of the commands just aren't compatible. This is fine in a normal project, but our weird extra scripting (scripts/localdev.sh) makes this painful. Additionally, a lot of this scripting is there to make things like test/fakemount work better: file ownership changes, file time modifications, full "nuke and pave" of the mounted dirs. But those are operating on the host filesystem, not podman volumes. This makes it very painful at times when using rootless podman, because it's a lot less forgiving of things like a deleted mount or permission manipulations (or at least I am finding it confusing to make it work). Finally, it's just gotten difficult to pick up after several months away - the code is actually easier to re-learn than the compose setup. The scripts and compose override and various hacks that were done to make dev easy... actually make it a lot tougher when anything goes wrong.

So it's time to redo the compose setup. This is already potentially complicated because a production setup is so different from dev: in production you'd only run the NCA services, but in dev you need all the ONI services (twice) and the ONI agents. Worst-case, we need a dev setup that is just more standardized and less scripted.

• Step 1: create a "build" image for all the Go pieces (we sort of have this already in the multi-stage docker/Dockerfile-app). Images should copy binaries in at the end of the Dockerfile so rebuilds are fast.
• Step 1b: investigate also having a "build" service, not just a temporary image. Combined with a shared "binaries" volume, I think the build service could be configured to watch for *.go changes and run rebuilds, populating the "binaries" volume, and all we'd need to do is a quick podman compose restart <service> command to get the new binaries running for the web server and job runners.
• Step 2: burn down all "local dev helper" script stuff.
  • Most of this stuff needs to be in the containers, not the host, and we might want a dedicated test container for any "helper" commands to keep the main service containers clean.
  • It might be better to just burn it all down and only build things up as we re-learn the needs of development.
  • It might also be that we really only need a handful of commands for copying issue sources into the right volumes - basically a modified test/copy-sources.go.
• Step 3: Massive test refactor.
  • Right now the test "recipes" all assume hybrid development, preventing me from testing anything in a podman environment.
  • The tests need a proper fully encapsulated sandbox that's 100% separated from the rest of the stack. Since podman still doesn't have consistent support for a top-level name field, this means either a custom command for tests (using -p for the project name) or custom service / volume names. This could get ugly.
  • The end-to-end tests will need a lot of investigation. Today's "test suite" relies on various functions in scripts/localdev.sh, which makes it tough to know what's in localdev.sh for test support versus dev support. prep_for_testing, for instance, is almost never needed in local development, but some of the functions it calls are important for both.
  • If necessary, end-to-end tests can go back to being a described process instead of a semi-automated set of commands.

[jechols]

I think it's worth ditching ONI from the dev stack. Let devs stand up their own ONI instances however they see fit, don't try to shoehorn it into NCA's stack. Then we could have fake agents that just pretend to talk to ONI and return success, or let devs use real agents for heavier dev work.