docs: sync with repo — CTE chimod chain, env vars, default compose - #18
Open
lukemartinlogan wants to merge 1 commit into
Open
docs: sync with repo — CTE chimod chain, env vars, default compose#18lukemartinlogan wants to merge 1 commit into
lukemartinlogan wants to merge 1 commit into
Conversation
Compare docs against the current source and fix what drifted.
New page (sdk/context-transfer-engine/chimod-chain.md): the CTE
interposition chain — cache(563) -> indexer(564) -> [compressor(562)] ->
replication(561) -> core(512). Covers what an interposer is (speaks the
core's method ids, overrides a few data verbs, forwards the rest via
next_pool_id, so a plain core::Client works unchanged), then each layer:
- replication: write-through to a REPLICA_FIXED|REPLICA_PERSISTENT set,
async sweep (replicate_period_ms, 0 = synchronous), replica-served
reads with primary re-cache, replica_score as the drop threshold,
ReplicateBlob/FlushTag.
- cache: why the copy is raw (keeps the zero-IPC SHM read path alive),
writer-local routing, the write-local-then-register coherence
protocol, speculative-copy verification, why the owner keeps no copy,
min_score as a floor.
- indexer: O(1) coalesced enqueue off the ack path, sweep vs lazy, the
read-your-writes drain before search, BM25 over the matched slice,
snapshot+WAL persistence vs in-memory, scope regexes, ReindexScan,
CLIO_INDEXER_PASSIVE.
Environment variables: add a full reference section to
deployment/configuration.md — startup, client/transport, SHM ingest
tuning (CLIO_SHM_IN_SHARDS/_ASYNC_SEND/_CLIENT_SPIN_US), CTE, CFS
adapters, bdev stats, task scheduling, logging, and the ADIOS2
large-scale init-stagger vars. CLIO_IPC_MODE was documented as "TCP
default"; unset actually auto-probes SHM -> IPC -> TCP.
Default compose: the docs showed a 3-module compose. Replace with the
shipped clio_default.yaml (CAE at its own pool 400.0 forwarding to CTE,
a persistent disk tier with persistence_level, metadata_log_path, and
the full chain), document the new config keys (learning_rate,
main/metadata_segment_size, conf_dir, gpu and swim sections,
existing_pool_id, gpu_metadata_cache, CAE LLM labeling), and refresh
both docker-compose examples, including a volume for ~/.clio/ now that
the default config writes persistent state there.
Deprecation notes were wrong in a dangerous direction: they claimed the
CHI_* env vars, <chimaera/...>/<hermes_shm/...> headers,
hshm::/hipc::/HSHM_*/CHI_* macros, ~/.chimaera/ config paths and the
chimaera CLI symlink still worked. None do — GetCompat() reads only
CLIO_<suffix>, the shim trees are gone, and all config-lookup candidates
are ~/.clio/clio.yaml. A script still exporting CHI_SERVER_CONF starts
successfully on the wrong config. Rewrite around what was removed, make
the migration sweep mandatory, and keep the two aliases that survive
(clio_cae_omni, the nested `clio_run runtime ...` forms).
CMake naming was stale across six pages: real targets are
clio::run::admin_client / clio_cte::core_client, the umbrella package is
find_package(clio-core CONFIG REQUIRED), the helper is
ClioCoreCommon.cmake, and the runtime define is CLIO_RUNTIME=1.
Also fix three pre-existing broken TOC anchors in the module dev guide
and add sidebar_position/frontmatter to the CTE pages. Site builds with
no broken links or anchors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compared the docs against the current state of
iowarp/clio-coreand fixed what had drifted.New page: Cache / Replication / Indexing ChiMods
docs/sdk/context-transfer-engine/chimod-chain.mddocuments the CTE interposition chain:It explains what an interposer is — a pool that speaks the CTE core's own method ids and task structs, overrides a handful of data verbs, and forwards everything else to
next_pool_id, so a plainclio::cte::core::Clientworks unchanged — and then each layer:REPLICA_FIXED | REPLICA_PERSISTENTset, the async sweep (replicate_period_ms,0= synchronous), replica-served reads with primary re-cache,replica_scoredoubling as the drop threshold, plusReplicateBlob/FlushTag.min_scoreas a score floor.ReindexScan, andCLIO_INDEXER_PASSIVE.Plus addressing (
CLIO_CTE_POOL), compose ordering rules, and a trim-the-chain table.Environment variables
Added a full reference section to
deployment/configuration.md: startup, client/transport, SHM ingest tuning (CLIO_SHM_IN_SHARDS/_ASYNC_SEND/_CLIENT_SPIN_US), CTE, CFS adapters, bdev stats, task scheduling, logging, and the ADIOS2 large-scale init-stagger vars.Corrected
CLIO_IPC_MODEin configuration.md and quick-start.mdx — it was documented as "TCP default", but unset actually auto-probes SHM → IPC → TCP.Default compose file
The docs showed a three-module compose. Replaced with the actually-shipped
clio_default.yaml: CAE at its own pool400.0forwarding to CTE, a persistent disk tier withpersistence_level,performance.metadata_log_path, and the full chain.Also documented config keys that had no coverage:
learning_rate,main_segment_size,metadata_segment_size,conf_dir, thegpuandswimsections,existing_pool_id/existing_pool_module,gpu_metadata_cache, and CAE transparent LLM labeling.Refreshed both Docker Compose examples, including a volume for
~/.clio/— the default config now writes the persistent tier, metadata log, and search index there, so without one they vanish ondocker compose down.Two things worth a closer look
1.
deprecation-notes.mdwas wrong in a dangerous direction. It claimed theCHI_*env vars,<chimaera/…>/<hermes_shm/…>headers, thehshm::/hipc::/HSHM_*/CHI_*macros, the~/.chimaera/config paths, and thechimaeraCLI symlink all still worked as aliases. None of them do —GetCompat()now reads onlyCLIO_<suffix>, the shim trees are gone, and all four config-lookup candidates are~/.clio/clio.yaml.The failure mode is quiet: an unset env var is not an error, so a launch script still exporting
CHI_SERVER_CONForCHI_PORTstarts the runtime successfully on the wrong configuration. Rewrote the page around what was removed, made the migration sweep mandatory rather than optional, and kept the two aliases that genuinely survive (clio_cae_omni, and the nestedclio_run runtime …/clio_run repo refreshforms).2. CMake naming was stale across six pages. The real targets are
clio::run::admin_client/clio_cte::core_client, the umbrella package isfind_package(clio-core CONFIG REQUIRED), the helper isClioCoreCommon.cmake, and the runtime define isCLIO_RUNTIME=1(notCHIMAERA_RUNTIME=1). Fixed in the module dev guide, the three base-module pages,cte.md, the scheduler guide, monitoring, hpc-cluster, and the module test guide.Also
sidebar_position/ frontmatter to the CTE pages so the section orders sensibly.Verification
npx docusaurus buildcompletes with no broken links and no broken anchors. Every claim above was checked against the source inclio-core(task/config headers,clio_default.yaml,config_manager.cc,ipc_manager.cc,ClioCoreCommon.cmake, the Dockerfiles) rather than inferred.🤖 Generated with Claude Code