fix: race condition causing builds stuck queued

[rocketeerbkw]

Checklist

• Affected Issues have been mentioned in the Closing issues section
• Documentation has been written/updated
• PR title is ready for changelog and subsystem label(s) applied

Attempts to fix a race condition that can leave a build stuck queued until the remote-controller is restarted. Seen in production but not able to reproduce locally.

The race condition:

1. New build comes in, picked up by build handler
2. Build CRD is created (direct k8s api) and added to BuildQueueCache
3. Reconciler/Queue system sees pending build in BuildQueueCache and attempts to load CRD (hits k8s cache only)
4. CRD is not in the cache, so returns not found
5. Error condition leaves build in bad state in queue cache, and never picked up again

Related error thrown in remote-controller:

ERROR v1beta2.LagoonBuild error running queue processor {"lagoonbuild":{"name":"lagoon-build-abcdef","namespace":"drupal-example-main"},"error":"LagoonBuild.crd.lagoon.sh \"lagoon-build-ghijkl\" not found"}

github.com/uselagoon/remote-controller/internal/controllers/v1beta2.(*LagoonBuildReconciler).whichBuildNext
    /workspace/internal/controllers/v1beta2/build_qoshandler.go:67
github.com/uselagoon/remote-controller/internal/controllers/v1beta2.(*LagoonBuildReconciler).qosBuildProcessor
    /workspace/internal/controllers/v1beta2/build_qoshandler.go:40
github.com/uselagoon/remote-controller/internal/controllers/v1beta2.(*LagoonBuildReconciler).Reconcile
    /workspace/internal/controllers/v1beta2/build_controller.go:160
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313

Closing issues

n/a

[shreddedbacon]

Simple enough, will see if it helps