Add Fake (in-memory) adapter - #24
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements the full Redis adapter functionality with in-process storage: waiting/active/stalled/delayed/completed/failed queues, signals, locks, maintenance, retries, repeatable jobs and dumps. Storage is shared between adapter instances with the same prefix, so multiple brokers in the same process can communicate (like tests do via Redis). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- test/utils.js exports adapterType/transporterType (WF_TEST_ADAPTER env var) - new 'test:integration:fake' npm script runs the whole suite without Redis - adapters.spec: add Fake adapter resolution cases, fix unawaited cleanup/stop - events.spec: wait for job event delivery before asserting - repeat.spec: use greaterThanOrEqual for child job createdAt (same-ms jobs) - batch.spec: relax multi-worker distribution range for the Fake adapter Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- README: Fake adapter section with usage examples and options table - CHANGELOG: Unreleased entry - CI: run the test suite with both Redis and Fake adapters - Add missing 'globals' devDependency (imported by eslint.config.mjs) - Fix no-useless-assignment lint error in adapters/index.ts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Fake CI matrix job has no Redis container, so the Redis-based cases in adapters.spec.ts hung on connect and failed the whole file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The '{ type: Adapters.Fake }' form silently fell back to the Redis
adapter, because 'opt.type instanceof BaseAdapter' is never true for a
class (only for an instance). The Redis class test only passed by
accident as the fallback is Redis. On CI (Fake matrix job, no Redis)
the Fake class case hung on connecting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new Fake (in-memory) adapter to the Moleculer workflows middleware, intended to enable local development and testing without requiring a Redis server, while keeping behavior largely aligned with the existing Redis adapter.
Changes:
- Introduces
FakeAdapter+FakeStorageto emulate Redis-like data structures and Pub/Sub semantics in-process. - Refactors shared adapter logic by moving
serializeJob,deserializeJob,getBackoffTime, andformatZrangeResultToObjectintoBaseAdapter. - Parameterizes integration tests and CI to run the suite against both
RedisandFakeadapters viaWF_TEST_ADAPTER.
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/utils.js | Adds adapterType/transporterType helpers driven by WF_TEST_ADAPTER. |
| test/unit/index.spec.ts | Enables Fake adapter resolver tests and adds class-based resolution coverage. |
| test/unit/fake.spec.ts | Adds unit tests for FakeAdapter keying, shared storage, storage semantics, and waiter wake/timeout behavior. |
| test/integration/timeout.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/stalled.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/signal.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/serialization.spec.ts | Uses adapterType in adapter config object. |
| test/integration/retries.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/retention.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/repeat.spec.ts | Uses adapterType and relaxes a timestamp assertion for adapter parity. |
| test/integration/middleware.spec.ts | Uses adapterType for both broker and worker. |
| test/integration/index.spec.ts | Uses adapterType and transporterType for multi-broker tests. |
| test/integration/events.spec.ts | Uses adapterType/transporterType and adds delays to allow async event delivery. |
| test/integration/delayed.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/collision.spec.ts | Uses adapterType instead of hard-coded "Redis". |
| test/integration/batch.spec.ts | Uses adapterType and adjusts distribution floor for Fake adapter behavior. |
| test/integration/adapters.spec.ts | Skips Redis-only cases under Fake and adds Fake adapter coverage; fixes cleanup awaiting. |
| src/adapters/redis.ts | Removes now-shared helper methods (moved into BaseAdapter); minor typing tweak for serializer. |
| src/adapters/index.ts | Registers Fake adapter and extends resolver to accept adapter classes. |
| src/adapters/fake.ts | Adds the new in-memory adapter implementation. |
| src/adapters/base.ts | Adds shared job (de)serialization, backoff computation, and zrange formatting utilities. |
| README.md | Documents Fake adapter usage and options with a non-production warning. |
| package.json | Adds test:integration:fake script and dev dependencies (cross-env, globals). |
| package-lock.json | Locks new dev dependencies. |
| CHANGELOG.md | Adds Unreleased entry describing Fake adapter and shared-method refactor. |
| .github/workflows/test.yml | Runs CI matrix for both adapters; skips Redis containers for Fake runs. |
Comment on lines
+639
to
+650
| // Wait for a new waiting job or the drain delay | ||
| await new Promise<void>(resolve => { | ||
| const waiter = () => { | ||
| clearTimeout(timer); | ||
| resolve(); | ||
| }; | ||
| const timer = setTimeout(() => { | ||
| this.storage.removeWaiter(waitingKey, waiter); | ||
| resolve(); | ||
| }, timeout * 1000); | ||
| this.storage.addWaiter(waitingKey, waiter); | ||
| }); |
Comment on lines
26
to
29
| | { | ||
| type: keyof typeof Adapters | typeof BaseAdapter; | ||
| options: BaseDefaultOptions | RedisAdapterOptions; | ||
| options: BaseDefaultOptions | FakeAdapterOptions | RedisAdapterOptions; | ||
| }; |
Comment on lines
+58
to
62
| if (typeof opt.type === "function") { | ||
| // Adapter class (e.g. Adapters.Fake) | ||
| AdapterClass = opt.type; | ||
| } else if (typeof opt.type === "string") { | ||
| AdapterClass = getByName(opt.type || "Redis"); |
- ResolvableAdapterType: 'options' is now optional, matching the resolve() behavior - resolve() throws a clear ServiceSchemaError when the given adapter class does not extend BaseAdapter, instead of instantiating any callable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lock() threw a plain WorkflowError, but processJob() only treats WorkflowAlreadyLocked as a non-failure case, so the skip branch could never trigger and lock contention always moved the job to the failed queue. Fixed in both the Redis and the Fake adapters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new Fake (in-memory) adapter that covers the same functionality as the Redis adapter, but stores jobs only in process memory. It is intended for testing & development, so no Redis server is needed. (The README feature list already advertised it, and the commented-out placeholders in
adapters/index.tsand the unit tests are now live.)Implementation
src/adapters/fake.ts—FakeAdapter extends BaseAdapter+ aFakeStorageengine that mimics the Redis data structures (hash, list, set, sorted set, string with lazy expiration).FakeAdapter.clearAll()is available for tests; data survivesdisconnect()like Redis data does.EventEmitter(signals,finished, delayedpromoteAtchannels).BRPOPLPUSHis replaced by a FIFO waiter queue: each pushed job wakes exactly one blocked worker (like Redis serves blocked clients), keepingdrainDelaysemantics.serializeJob,deserializeJob,getBackoffTimeandformatZrangeResultToObjectmoved fromRedisAdaptertoBaseAdapter(shared with the new adapter).test/unit/fake.spec.ts(39 tests: key scheme parity, shared/isolated storages, storage command semantics, lazy lock/signal expiration,popWaitingJobwake/timeout).WF_TEST_ADAPTER=Fake npm run test:integration(or the newtest:integration:fakescript) runs all 78 integration tests without Redis (the multi-broker specs switch to the Fake transporter as well).RedisandFakeadapters.### Fake adaptersection with usage examples, options table and a 'not for production' warning; CHANGELOG entry.Intentional behavioral differences vs. Redis adapter
In-memory execution makes some races deterministic that network latency hides on Redis:
broker.wf.remove) while it is being processed, the Fake adapter does not re-create the job hash when persistingstartedAt/result/error, so the job stays removed.cleanUp()really deletes everything with the prefix. Note: the Redis adapter has a bug here — it scansthis.prefix + ":*", but the prefix already ends with:, so the pattern never matches. Worth a separate fix.batch.spec.tsuses a lower floor for Fake (850 vs 980) — single-threaded scheduling is slightly less even.Other notes
lock()throws a plainWorkflowError, whileprocessJobcatchesWorkflowAlreadyLocked, so that catch branch never triggers.adapters.spec.ts(unawaitedcleanup()/broker.stop(), cleanup used the wrong workflow nameadapters.simple).globalsdevDependency (imported byeslint.config.mjsbut not declared — lint could not run without it).Test plan
npm run check+npm run lintclean (only pre-existing warnings)npm run test:unit— 58 passedWF_TEST_ADAPTER=Fakeintegration suite — 78/78 passed, no Redis needed🤖 Generated with Claude Code