docs: rewrite README as concise landing page and add architecture diagrams [WIP] - #161
Draft
coketaste wants to merge 10 commits into
Draft
docs: rewrite README as concise landing page and add architecture diagrams [WIP]#161coketaste wants to merge 10 commits into
coketaste wants to merge 10 commits into
Conversation
…ures Trim root README from 707 to 258 lines by moving deep reference material (profiling tables, extended config/usage recipes, tips) into docs/ and linking out. Replace the ASCII architecture block and the stale, unreferenced docs/img PNGs with accurate inline Mermaid figures for the layered architecture, build->run->report pipeline, and deployment-target inference. Move the parallelism/infrastructure capability matrices into docs/launchers.md and add Mermaid workflow diagrams to docs/deployment.md and docs/README.md. Co-Authored-By: Claude <noreply@anthropic.com>
…t key The launcher field must be "megatron-lm" per VALID_LAUNCHERS in deployment/common.py — "megatron" silently falls through to the Docker fallback on SLURM. Also removes distributed.master_port, which is never read (SLURM reads distributed.port; K8s reads a separate top-level launcher.master_port object), across all affected example configs.
…sage Corrects --csv-file to the real --csv-file-path/--file flags, fills in missing `database` command flags (--unique-key/-k, --batch-size, --no-upsert, --no-index, --dry-run, MONGO_AUTH_SOURCE/MONGO_TIMEOUT_MS), fixes wrong `run --output`/`--tools-config` defaults, fixes broken space-separated --tags syntax, and updates the stale version badge.
Removes fabricated timeout_multiplier/service_account keys and the vllm.tensor_parallel_size example (never read; real value derives from distributed.nproc_per_node). Fixes SLURM gpus_per_node default (1 -> 8), docker_env_vars.MAD_ROCM_PATH -> ROCM_PATH, the Data Provider schema (fabricated data_sources wrapper -> real flat shape), and credential key names (AMD_GITHUB -> PUBLIC_GITHUB_ROCM_KEY, uppercase USERNAME/PASSWORD).
Corrects the backwards/incomplete K8s "Configuration Priority" list to match config_loader.py's real 5-layer merge order (including the previously-missing AMD multi-GPU vendor layer), and removes nonexistent SLURM fields (mem, mail_user, mail_type). In examples/k8s-configs and examples/slurm-configs READMEs: fixes ~35 stale/renamed file paths, removes dead Primus example references, marks gpu_vendor/guest_os/deploy as non-required with real defaults, removes the invalid "RHEL" guest_os value, documents results_pvc and host_ipc as non-configurable (host_ipc previously told users to set a key that is always silently overridden), adds missing results_storage_size / data_storage_size / allow_privileged_profiling fields, and documents the SLURM node-health preflight keys (enable_node_check, auto_cleanup_nodes, allow_submit_without_clean_nodes, verbose_node_check).
…ples Fixes "megatron" -> "megatron-lm" launcher value, removes the nonexistent --config flag in favor of --additional-context-file, clarifies the dead distributed.master_port key (SLURM reads distributed.port; K8s reads a separate top-level launcher.master_port object), corrects the 5-node auto-split table row (1/3, not 2/2), fixes the vLLM SLURM multi-node description (data-parallel, not TP+PP with Ray), and removes broken Primus example links.
…ample Removes the nonexistent --tools therock_check flag, fixes the "Default Tool Configuration" example (rocprof command, gpu_info_power_profiler env vars using bare names instead of the real POWER_/VRAM_-prefixed names) across all Multi-GPU and sampling-rate examples, and adds the undocumented tool names (rocprof_hip_only, rocprof_sys, rocprofv3, rocprofv3_agent, rocprofv3_agent_counter, hipblaslt_trace, instruction_mix.txt).
… refs Notes that deployment_config in build_manifest.json is only written for non-local deployments (per _save_deployment_config in build_orchestrator.py), and adds the always-present context/ credentials_required keys to the example. Removes a duplicated `madengine --version` line in installation.md and fixes a stale test path in contributing.md (tests/test_cli.py -> tests/unit/test_cli.py).
The README described the module as "Not yet implemented" and documented a fictional future API (mongodb_client.py/MongoDBClient, local_storage.py/LocalStorage, api.py/ingest_results()), even though mongodb.py is fully implemented and wired into the `database` CLI command. Rewrites the README around the real classes (MongoDBConfig, UploadOptions, UploadResult, DocumentLoader/JSONLoader/CSVLoader, DocumentTransformer, MongoDBUploader, upload_file_to_mongodb) and adds a CLI-flag-to-API-param mapping table.
execution/README.md: fixes build_all_models's models_list -> models param, run_container's fabricated model_docker/gpu_ids params -> real docker_image string param, wrong status value casing/set (successful/failed/timeout -> SUCCESS/FAILURE/SKIPPED), wrong result key (duration -> test_duration), and documents the previously-missing dockerfile_utils.py and container_runner_helpers.py files. reporting/README.md: removes the fabricated "Legacy Reporting Tools" section claiming csv_to_html.py/csv_to_email.py live in a nonexistent tools/ directory and are unused by the modern CLI (they live in reporting/ and back `report to-html`/`report to-email`), fixes the update_perf_csv()/flatten_tags() example signatures, documents the perf_entry.csv/.json side effect, and adds a missing entry for update_perf_super.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
README.mdas a concise landing page, replacing the long-form content with a tighter overview (734 → much shorter, ~460 net removed)docs/README.md(CLI → orchestration → execution/deployment → launcher flow) anddocs/deployment.md(build/deploy workflow and target-inferencedecision tree), replacing ASCII-art diagrams
docs/launchers.md: per-launcher parallelism support (TP/PP/DP/CP/FSDP/EP) and per-target infrastructure features (local/K8s/SLURM)docs/img/architecture_overview.pnganddocs/img/distributed_workflow.png(superseded by inline Mermaid diagrams)Test plan
README.md,docs/README.md,docs/deployment.md, anddocs/launchers.mdon GitHub (or a Markdown/Mermaid previewer) to confirm diagrams and tables render correctly