Skip to content

docs: sync with repo — CTE chimod chain, env vars, default compose - #18

Open
lukemartinlogan wants to merge 1 commit into
mainfrom
docs/cte-chimod-chain-and-env-vars
Open

docs: sync with repo — CTE chimod chain, env vars, default compose#18
lukemartinlogan wants to merge 1 commit into
mainfrom
docs/cte-chimod-chain-and-env-vars

Conversation

@lukemartinlogan

Copy link
Copy Markdown
Contributor

Compared the docs against the current state of iowarp/clio-core and fixed what had drifted.

New page: Cache / Replication / Indexing ChiMods

docs/sdk/context-transfer-engine/chimod-chain.md documents the CTE interposition chain:

cache(563.0) → indexer(564.0) → [compressor(562.0) →] replication(561.0) → core(512.0)

It explains what an interposer is — a pool that speaks the CTE core's own method ids and task structs, overrides a handful of data verbs, and forwards everything else to next_pool_id, so a plain clio::cte::core::Client works unchanged — and then each layer:

  • Replication — write-through to a fixed REPLICA_FIXED | REPLICA_PERSISTENT set, the async sweep (replicate_period_ms, 0 = synchronous), replica-served reads with primary re-cache, replica_score doubling as the drop threshold, plus ReplicateBlob / FlushTag.
  • Cache — why the copy is stored raw (it's what keeps the zero-IPC SHM read path alive for compressed blobs), writer-local routing, the write-local-then-register coherence protocol, speculative-copy verification, why the owner node deliberately keeps no copy, and min_score as a score floor.
  • Indexer — indexing is off the ack path (O(1) coalesced enqueue), sweep vs. lazy, the read-your-writes drain before every search, BM25 over the matched slice only, snapshot+WAL persistence vs. in-memory, scope regexes, ReindexScan, and CLIO_INDEXER_PASSIVE.

Plus addressing (CLIO_CTE_POOL), compose ordering rules, and a trim-the-chain table.

Environment variables

Added a full reference section to deployment/configuration.md: startup, client/transport, SHM ingest tuning (CLIO_SHM_IN_SHARDS / _ASYNC_SEND / _CLIENT_SPIN_US), CTE, CFS adapters, bdev stats, task scheduling, logging, and the ADIOS2 large-scale init-stagger vars.

Corrected CLIO_IPC_MODE in configuration.md and quick-start.mdx — it was documented as "TCP default", but unset actually auto-probes SHM → IPC → TCP.

Default compose file

The docs showed a three-module compose. Replaced with the actually-shipped clio_default.yaml: CAE at its own pool 400.0 forwarding to CTE, a persistent disk tier with persistence_level, performance.metadata_log_path, and the full chain.

Also documented config keys that had no coverage: learning_rate, main_segment_size, metadata_segment_size, conf_dir, the gpu and swim sections, existing_pool_id / existing_pool_module, gpu_metadata_cache, and CAE transparent LLM labeling.

Refreshed both Docker Compose examples, including a volume for ~/.clio/ — the default config now writes the persistent tier, metadata log, and search index there, so without one they vanish on docker compose down.

Two things worth a closer look

1. deprecation-notes.md was wrong in a dangerous direction. It claimed the CHI_* env vars, <chimaera/…> / <hermes_shm/…> headers, the hshm:: / hipc:: / HSHM_* / CHI_* macros, the ~/.chimaera/ config paths, and the chimaera CLI symlink all still worked as aliases. None of them do — GetCompat() now reads only CLIO_<suffix>, the shim trees are gone, and all four config-lookup candidates are ~/.clio/clio.yaml.

The failure mode is quiet: an unset env var is not an error, so a launch script still exporting CHI_SERVER_CONF or CHI_PORT starts the runtime successfully on the wrong configuration. Rewrote the page around what was removed, made the migration sweep mandatory rather than optional, and kept the two aliases that genuinely survive (clio_cae_omni, and the nested clio_run runtime … / clio_run repo refresh forms).

2. CMake naming was stale across six pages. The real targets are clio::run::admin_client / clio_cte::core_client, the umbrella package is find_package(clio-core CONFIG REQUIRED), the helper is ClioCoreCommon.cmake, and the runtime define is CLIO_RUNTIME=1 (not CHIMAERA_RUNTIME=1). Fixed in the module dev guide, the three base-module pages, cte.md, the scheduler guide, monitoring, hpc-cluster, and the module test guide.

Also

  • Fixed three pre-existing broken TOC anchors in the module dev guide.
  • Added sidebar_position / frontmatter to the CTE pages so the section orders sensibly.

Verification

npx docusaurus build completes with no broken links and no broken anchors. Every claim above was checked against the source in clio-core (task/config headers, clio_default.yaml, config_manager.cc, ipc_manager.cc, ClioCoreCommon.cmake, the Dockerfiles) rather than inferred.

🤖 Generated with Claude Code

Compare docs against the current source and fix what drifted.

New page (sdk/context-transfer-engine/chimod-chain.md): the CTE
interposition chain — cache(563) -> indexer(564) -> [compressor(562)] ->
replication(561) -> core(512). Covers what an interposer is (speaks the
core's method ids, overrides a few data verbs, forwards the rest via
next_pool_id, so a plain core::Client works unchanged), then each layer:

  - replication: write-through to a REPLICA_FIXED|REPLICA_PERSISTENT set,
    async sweep (replicate_period_ms, 0 = synchronous), replica-served
    reads with primary re-cache, replica_score as the drop threshold,
    ReplicateBlob/FlushTag.
  - cache: why the copy is raw (keeps the zero-IPC SHM read path alive),
    writer-local routing, the write-local-then-register coherence
    protocol, speculative-copy verification, why the owner keeps no copy,
    min_score as a floor.
  - indexer: O(1) coalesced enqueue off the ack path, sweep vs lazy, the
    read-your-writes drain before search, BM25 over the matched slice,
    snapshot+WAL persistence vs in-memory, scope regexes, ReindexScan,
    CLIO_INDEXER_PASSIVE.

Environment variables: add a full reference section to
deployment/configuration.md — startup, client/transport, SHM ingest
tuning (CLIO_SHM_IN_SHARDS/_ASYNC_SEND/_CLIENT_SPIN_US), CTE, CFS
adapters, bdev stats, task scheduling, logging, and the ADIOS2
large-scale init-stagger vars. CLIO_IPC_MODE was documented as "TCP
default"; unset actually auto-probes SHM -> IPC -> TCP.

Default compose: the docs showed a 3-module compose. Replace with the
shipped clio_default.yaml (CAE at its own pool 400.0 forwarding to CTE,
a persistent disk tier with persistence_level, metadata_log_path, and
the full chain), document the new config keys (learning_rate,
main/metadata_segment_size, conf_dir, gpu and swim sections,
existing_pool_id, gpu_metadata_cache, CAE LLM labeling), and refresh
both docker-compose examples, including a volume for ~/.clio/ now that
the default config writes persistent state there.

Deprecation notes were wrong in a dangerous direction: they claimed the
CHI_* env vars, <chimaera/...>/<hermes_shm/...> headers,
hshm::/hipc::/HSHM_*/CHI_* macros, ~/.chimaera/ config paths and the
chimaera CLI symlink still worked. None do — GetCompat() reads only
CLIO_<suffix>, the shim trees are gone, and all config-lookup candidates
are ~/.clio/clio.yaml. A script still exporting CHI_SERVER_CONF starts
successfully on the wrong config. Rewrite around what was removed, make
the migration sweep mandatory, and keep the two aliases that survive
(clio_cae_omni, the nested `clio_run runtime ...` forms).

CMake naming was stale across six pages: real targets are
clio::run::admin_client / clio_cte::core_client, the umbrella package is
find_package(clio-core CONFIG REQUIRED), the helper is
ClioCoreCommon.cmake, and the runtime define is CLIO_RUNTIME=1.

Also fix three pre-existing broken TOC anchors in the module dev guide
and add sidebar_position/frontmatter to the CTE pages. Site builds with
no broken links or anchors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant