Skip to content

Add Fake (in-memory) adapter - #24

Merged
icebob merged 8 commits into
masterfrom
feature/fake-adapter
Jul 12, 2026
Merged

Add Fake (in-memory) adapter#24
icebob merged 8 commits into
masterfrom
feature/fake-adapter

Conversation

@icebob

@icebob icebob commented Jul 12, 2026

Copy link
Copy Markdown
Member

Summary

Adds a new Fake (in-memory) adapter that covers the same functionality as the Redis adapter, but stores jobs only in process memory. It is intended for testing & development, so no Redis server is needed. (The README feature list already advertised it, and the commented-out placeholders in adapters/index.ts and the unit tests are now live.)

Implementation

  • src/adapters/fake.tsFakeAdapter extends BaseAdapter + a FakeStorage engine that mimics the Redis data structures (hash, list, set, sorted set, string with lazy expiration).
    • The storage is a static registry keyed by prefix, so all adapter instances (workers + middleware clients, even across multiple brokers in one process) share data — the same way tests communicate through a common Redis server. FakeAdapter.clearAll() is available for tests; data survives disconnect() like Redis data does.
    • Pub/Sub is replaced by an EventEmitter (signals, finished, delayed promoteAt channels).
    • The blocking BRPOPLPUSH is replaced by a FIFO waiter queue: each pushed job wakes exactly one blocked worker (like Redis serves blocked clients), keeping drainDelay semantics.
    • Job data round-trips through the Moleculer serializer (like Redis), so payload mutation / non-serializable bugs surface in tests too.
  • Refactor: serializeJob, deserializeJob, getBackoffTime and formatZrangeResultToObject moved from RedisAdapter to BaseAdapter (shared with the new adapter).
  • Tests:
    • New test/unit/fake.spec.ts (39 tests: key scheme parity, shared/isolated storages, storage command semantics, lazy lock/signal expiration, popWaitingJob wake/timeout).
    • The whole integration suite is parametrized over the adapter: WF_TEST_ADAPTER=Fake npm run test:integration (or the new test:integration:fake script) runs all 78 integration tests without Redis (the multi-broker specs switch to the Fake transporter as well).
    • CI matrix now runs the suite with both Redis and Fake adapters.
  • Docs: README ### Fake adapter section with usage examples, options table and a 'not for production' warning; CHANGELOG entry.

Intentional behavioral differences vs. Redis adapter

In-memory execution makes some races deterministic that network latency hides on Redis:

  • If a job is removed (broker.wf.remove) while it is being processed, the Fake adapter does not re-create the job hash when persisting startedAt/result/error, so the job stays removed.
  • The no-arg cleanUp() really deletes everything with the prefix. Note: the Redis adapter has a bug here — it scans this.prefix + ":*", but the prefix already ends with :, so the pattern never matches. Worth a separate fix.
  • The multi-worker distribution assertion in batch.spec.ts uses a lower floor for Fake (850 vs 980) — single-threaded scheduling is slightly less even.

Other notes

  • Replicated as-is for parity (possible upstream bug): lock() throws a plain WorkflowError, while processJob catches WorkflowAlreadyLocked, so that catch branch never triggers.
  • Fixed pre-existing issues in adapters.spec.ts (unawaited cleanup()/broker.stop(), cleanup used the wrong workflow name adapters.simple).
  • Added the missing globals devDependency (imported by eslint.config.mjs but not declared — lint could not run without it).

Test plan

  • npm run check + npm run lint clean (only pre-existing warnings)
  • npm run test:unit — 58 passed
  • WF_TEST_ADAPTER=Fake integration suite — 78/78 passed, no Redis needed
  • Redis integration suite — passed (the occasional single flaky failure under full parallel load exists on master too and passes standalone)

🤖 Generated with Claude Code

icebob and others added 6 commits July 12, 2026 20:59
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements the full Redis adapter functionality with in-process storage:
waiting/active/stalled/delayed/completed/failed queues, signals, locks,
maintenance, retries, repeatable jobs and dumps. Storage is shared between
adapter instances with the same prefix, so multiple brokers in the same
process can communicate (like tests do via Redis).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- test/utils.js exports adapterType/transporterType (WF_TEST_ADAPTER env var)
- new 'test:integration:fake' npm script runs the whole suite without Redis
- adapters.spec: add Fake adapter resolution cases, fix unawaited cleanup/stop
- events.spec: wait for job event delivery before asserting
- repeat.spec: use greaterThanOrEqual for child job createdAt (same-ms jobs)
- batch.spec: relax multi-worker distribution range for the Fake adapter

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- README: Fake adapter section with usage examples and options table
- CHANGELOG: Unreleased entry
- CI: run the test suite with both Redis and Fake adapters
- Add missing 'globals' devDependency (imported by eslint.config.mjs)
- Fix no-useless-assignment lint error in adapters/index.ts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Fake CI matrix job has no Redis container, so the Redis-based cases
in adapters.spec.ts hung on connect and failed the whole file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The '{ type: Adapters.Fake }' form silently fell back to the Redis
adapter, because 'opt.type instanceof BaseAdapter' is never true for a
class (only for an instance). The Redis class test only passed by
accident as the fallback is Redis. On CI (Fake matrix job, no Redis)
the Fake class case hung on connecting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new Fake (in-memory) adapter to the Moleculer workflows middleware, intended to enable local development and testing without requiring a Redis server, while keeping behavior largely aligned with the existing Redis adapter.

Changes:

  • Introduces FakeAdapter + FakeStorage to emulate Redis-like data structures and Pub/Sub semantics in-process.
  • Refactors shared adapter logic by moving serializeJob, deserializeJob, getBackoffTime, and formatZrangeResultToObject into BaseAdapter.
  • Parameterizes integration tests and CI to run the suite against both Redis and Fake adapters via WF_TEST_ADAPTER.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/utils.js Adds adapterType/transporterType helpers driven by WF_TEST_ADAPTER.
test/unit/index.spec.ts Enables Fake adapter resolver tests and adds class-based resolution coverage.
test/unit/fake.spec.ts Adds unit tests for FakeAdapter keying, shared storage, storage semantics, and waiter wake/timeout behavior.
test/integration/timeout.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/stalled.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/signal.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/serialization.spec.ts Uses adapterType in adapter config object.
test/integration/retries.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/retention.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/repeat.spec.ts Uses adapterType and relaxes a timestamp assertion for adapter parity.
test/integration/middleware.spec.ts Uses adapterType for both broker and worker.
test/integration/index.spec.ts Uses adapterType and transporterType for multi-broker tests.
test/integration/events.spec.ts Uses adapterType/transporterType and adds delays to allow async event delivery.
test/integration/delayed.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/collision.spec.ts Uses adapterType instead of hard-coded "Redis".
test/integration/batch.spec.ts Uses adapterType and adjusts distribution floor for Fake adapter behavior.
test/integration/adapters.spec.ts Skips Redis-only cases under Fake and adds Fake adapter coverage; fixes cleanup awaiting.
src/adapters/redis.ts Removes now-shared helper methods (moved into BaseAdapter); minor typing tweak for serializer.
src/adapters/index.ts Registers Fake adapter and extends resolver to accept adapter classes.
src/adapters/fake.ts Adds the new in-memory adapter implementation.
src/adapters/base.ts Adds shared job (de)serialization, backoff computation, and zrange formatting utilities.
README.md Documents Fake adapter usage and options with a non-production warning.
package.json Adds test:integration:fake script and dev dependencies (cross-env, globals).
package-lock.json Locks new dev dependencies.
CHANGELOG.md Adds Unreleased entry describing Fake adapter and shared-method refactor.
.github/workflows/test.yml Runs CI matrix for both adapters; skips Redis containers for Fake runs.

Comment thread src/adapters/fake.ts
Comment on lines +639 to +650
// Wait for a new waiting job or the drain delay
await new Promise<void>(resolve => {
const waiter = () => {
clearTimeout(timer);
resolve();
};
const timer = setTimeout(() => {
this.storage.removeWaiter(waitingKey, waiter);
resolve();
}, timeout * 1000);
this.storage.addWaiter(waitingKey, waiter);
});
Comment thread src/adapters/fake.ts Outdated
Comment thread src/adapters/index.ts
Comment on lines 26 to 29
| {
type: keyof typeof Adapters | typeof BaseAdapter;
options: BaseDefaultOptions | RedisAdapterOptions;
options: BaseDefaultOptions | FakeAdapterOptions | RedisAdapterOptions;
};
Comment thread src/adapters/index.ts
Comment on lines +58 to 62
if (typeof opt.type === "function") {
// Adapter class (e.g. Adapters.Fake)
AdapterClass = opt.type;
} else if (typeof opt.type === "string") {
AdapterClass = getByName(opt.type || "Redis");
icebob and others added 2 commits July 12, 2026 22:25
- ResolvableAdapterType: 'options' is now optional, matching the
  resolve() behavior
- resolve() throws a clear ServiceSchemaError when the given adapter
  class does not extend BaseAdapter, instead of instantiating any
  callable

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lock() threw a plain WorkflowError, but processJob() only treats
WorkflowAlreadyLocked as a non-failure case, so the skip branch could
never trigger and lock contention always moved the job to the failed
queue. Fixed in both the Redis and the Fake adapters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@icebob
icebob merged commit e874fa2 into master Jul 12, 2026
10 checks passed
@icebob
icebob deleted the feature/fake-adapter branch July 12, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants