feat: add duplicate family detection

README.md

@@ -27,6 +27,7 @@ Common workflows:
 ```bash
 npx codebase-intelligence hotspots ./src --metric complexity --limit 10
 npx codebase-intelligence opportunities ./src --limit 10
+npx codebase-intelligence duplicates ./src --mode mild --min-tokens 30
 npx codebase-intelligence impact ./src parseCodebase
 npx codebase-intelligence dead-exports ./src --limit 20
 npx codebase-intelligence changes ./src --json
@@ -56,7 +57,7 @@ claude mcp add -s user -t stdio codebase-intelligence -- npx -y codebase-intelli
 
 ## Features
 
-- **18 CLI commands** for architecture analysis, dependency impact, improvement opportunities, dead code detection, search, CI rules, and agent setup
+- **19 CLI commands** for architecture analysis, dependency impact, duplicate families, improvement opportunities, dead code detection, search, CI rules, and agent setup
 - **Machine-readable JSON output** (`--json`) for automation and CI pipelines
 - **Auto-cached index** in `.codebase-intelligence/` for fast repeat queries
 - **Cache migration facts** in JSON (`cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`)
@@ -66,7 +67,7 @@ claude mcp add -s user -t stdio codebase-intelligence -- npx -y codebase-intelli
 - **Process tracing** — detect entry points and execution flows through the call graph
 - **Community detection** — Louvain clustering for natural file groupings
 - **Agent adoption** — `init` writes per-agent instruction files + installs a skill so AI agents query CI before grep/read
-- **MCP parity (secondary)** — same analysis and rules gate available as 17 MCP tools, 2 prompts, and 3 resources
+- **MCP parity (secondary)** — same analysis and rules gate available as 18 MCP tools, 2 prompts, and 3 resources
 
 ## Installation
 
@@ -103,6 +104,7 @@ codebase-intelligence <command> <path> [options]
 | `forces` | Cohesion/tension/escape-velocity analysis |
 | `dead-exports` | Unused export detection |
 | `opportunities` | Ranked code-quality and refactoring opportunities |
+| `duplicates` | Duplicate function families (`strict`, `mild`, `weak`) |
 | `groups` | Top-level directory groups + aggregate metrics |
 | `symbol` | Callers/callees and symbol metrics |
 | `impact` | Symbol-level blast radius |
@@ -121,6 +123,10 @@ codebase-intelligence <command> <path> [options]
 | `--limit <n>` | Limit results on supported commands |
 | `--metric <m>` | Select ranking metric for `hotspots` |
 | `--scope <s>` | Select git diff scope for `changes`: `staged`, `unstaged`, `all` |
+| `--mode <m>` | Select clone mode for `duplicates`: `strict`, `mild`, `weak` |
+| `--min-tokens <n>` | Minimum duplicate token size for `duplicates` |
+| `--skip-local` | Ignore duplicate families confined to one file |
+| `--trace <id>` | Return token evidence for one duplicate family |
 
 The scanner always excludes common generated and agent-workspace directories such as `.codebase-intelligence/`, legacy `.code-visualizer/`, `.next/`, `dist/`, `coverage/`, `.worktrees/`, and `.claude/worktrees/`.
 

docs/architecture.md

@@ -7,7 +7,7 @@ CLI (commander)
   |
   v
 Parser (TS Compiler API)
-  | extracts: files, exports, symbols, type facts, imports, LOC, complexity, churn, test mapping
+  | extracts: files, exports, symbols, type facts, duplicate tokens, imports, LOC, complexity, churn, test mapping
   v
 Graph Builder (graphology)
   | creates: nodes (file + function), edges (imports with symbols/weights)
@@ -25,7 +25,7 @@ Core (shared computation)
   |     typed descriptors, input schemas, CLI/MCP adapters, result wrappers, text formatters
   v
 MCP (stdio)                    CLI (terminal/CI)
-  | 17 tools, 2 prompts,        | 18 commands with text + JSON
+  | 18 tools, 2 prompts,        | 19 commands with text + JSON
   | 3 resources for LLMs        | output for humans and CI
 ```
 
@@ -36,16 +36,18 @@ src/
   types/index.ts       <- ALL interfaces (single source of truth)
   parser/index.ts      <- Parse orchestration + imports/exports/call sites + git churn/test detection
   parser/type-facts.ts <- Type signatures, parameters, consumed/produced shape facts
+  parser/duplication.ts <- Function-body clone token extraction
   parser/symbols.ts    <- Symbol inventory + symbol complexity
   graph/index.ts       <- graphology graph + symbol/type graph + circular dep detection
   analyzer/index.ts    <- All metric computation
   graph-loader/index.ts <- Shared parse/build/analyze/cache pipeline + progress events
   core/index.ts        <- Shared result computation (MCP + CLI)
   operations/index.ts  <- Analysis operation descriptors + typed input schemas
   operations/formatters.ts <- Result-object text formatters for CLI commands
+  duplication/index.ts <- Duplicate family detection + trace evidence
   config/index.ts      <- Config discovery + zod validation
   rules/index.ts       <- Rules engine + registry (check command + MCP check tool)
-  mcp/index.ts         <- 17 MCP tools for LLM integration
+  mcp/index.ts         <- 18 MCP tools for LLM integration
   mcp/hints.ts         <- Operation-keyed next-step hints for MCP tool responses
   impact/index.ts      <- Symbol-level impact analysis + rename planning
   search/index.ts      <- BM25 search engine
@@ -67,7 +69,7 @@ loadCodebaseGraph(rootDir)
   -> otherwise emits progress events through parse/build/analyze/cache
 
 parseCodebase(rootDir)
-  -> ParsedFile[] (with churn, complexity, test mapping, symbol type facts)
+  -> ParsedFile[] (with churn, complexity, test mapping, symbol type facts, duplicate token facts)
 
 buildGraph(parsedFiles)
   -> BuiltGraph { graph: Graph, nodes: GraphNode[], edges: GraphEdge[] }
@@ -80,7 +82,7 @@ analyzeGraph(builtGraph, parsedFiles)
      }
 
 startMcpServer(codebaseGraph)
-  -> stdio MCP server with 17 tools, 2 prompts, 3 resources
+  -> stdio MCP server with 18 tools, 2 prompts, 3 resources
 
 runOperation(operation, codebaseGraph, input, context)
   -> { ok: true, data } | { ok: false, error, data? }
@@ -91,6 +93,7 @@ runOperation(operation, codebaseGraph, input, context)
 - **Dual interface**: MCP stdio for LLM agents, CLI subcommands for humans/CI. Both consume `src/core/`.
 - **Operation registry foundation**: Analysis operations now have typed descriptors in `src/operations/` with operation names, CLI command names, MCP tool names, input schemas, discriminated run results, and result-object text formatters. MCP tool registration and CLI command execution consume those descriptors; CLI JSON remains raw result data plus cache facts.
 - **Type/Shape facts**: Full-program parsing stores compact parameter/return/type-parameter facts on parsed symbols. `file`, `symbol`, and `search` JSON expose those facts additively; search indexes consumed/produced shape tokens so agents can ask which symbols touch a shape without a new command.
+- **Duplication families**: Parser stores deterministic function-body token streams on symbols. `duplicates` / `find_duplicates` groups symbols into strict, renamed, and near-miss clone families, with optional trace evidence for AI agents before refactors.
 - **Shared graph-load pipeline**: CLI commands and MCP stdio startup both use `src/graph-loader/` for path checks, legacy cache migration, cache reuse, parse/build/analyze, optional persistence, and stderr progress events.
 - **graphology**: In-memory graph with O(1) neighbor lookup. PageRank and betweenness computed via graphology-metrics.
 - **Batch git churn**: Single `git log --all --name-only` call, parsed for all files. Avoids O(n) subprocess spawning.

docs/cli-reference.md

@@ -1,6 +1,6 @@
 # CLI Reference
 
-18 commands for terminal and CI use. The 16 analysis commands have full parity with MCP tools and auto-cache the index to `.codebase-intelligence/`; `check` runs the rules gate; `init` sets up agent adoption.
+19 commands for terminal and CI use. The 17 analysis commands have full parity with MCP tools and auto-cache the index to `.codebase-intelligence/`; `check` runs the rules gate; `init` sets up agent adoption.
 
 ## Commands
 
@@ -106,6 +106,18 @@ codebase-intelligence opportunities <path> [--limit <n>] [--json] [--force]
 
 **Output:** ranked opportunities with kind, priority, confidence, score, target, evidence, and suggested follow-up commands.
 
+### duplicates
+
+Detect duplicate function families.
+
+```bash
+codebase-intelligence duplicates <path> [--mode <mode>] [--min-tokens <n>] [--skip-local] [--trace <id>] [--json] [--force]
+```
+
+**Modes:** `strict` preserves identifiers/literals, `mild` normalizes identifiers/literals for renamed clones, `weak` uses deterministic sequence similarity for near-miss clones.
+
+**Output:** duplicate family IDs, member symbols, token counts, similarity threshold/score, and optional trace evidence for one family.
+
 ### groups
 
 Top-level directory groups with aggregate metrics.
@@ -209,6 +221,10 @@ codebase-intelligence init [path] [--agents <list>] [--all] [--skill] [--gitigno
 | `--force` | All commands | Re-parse even if cached index matches HEAD |
 | `--metric <m>` | hotspots | Metric to rank by (default: coupling) |
 | `--limit <n>` | hotspots, search, dead-exports, opportunities, processes | Max results |
+| `--mode <m>` | duplicates | Clone mode: strict, mild, weak |
+| `--min-tokens <n>` | duplicates | Minimum function body token count (default: 30) |
+| `--skip-local` | duplicates | Ignore families confined to one file |
+| `--trace <id>` | duplicates | Return token evidence for one family id |
 | `--scope <s>` | changes | Git diff scope: staged, unstaged, all |
 | `--depth <n>` | dependents | Max traversal depth (default: 2) |
 | `--cohesion <n>` | forces | Min cohesion threshold (default: 0.6) |

docs/data-model.md

@@ -24,6 +24,7 @@ ParsedExport {
   isDefault: boolean
   complexity: number      // Cyclomatic complexity (branch count, min 1)
   typeFacts?: SymbolTypeFacts
+  duplication?: SymbolDuplicationFacts
 }
 
 ParsedSymbol extends ParsedExport {
@@ -40,6 +41,12 @@ SymbolTypeFacts {
   confidence: "resolved" | "syntax"
 }
 
+SymbolDuplicationFacts {
+  tokenCount: number
+  tokens: Record<"strict" | "mild" | "weak", string[]> // Function-body token streams
+  hashes: Record<"strict" | "mild" | "weak", string>  // Deterministic family grouping keys
+}
+
 ParsedImport {
   from: string            // Raw import path
   resolvedFrom: string    // Resolved relative path (after .js->.ts mapping)
@@ -79,6 +86,7 @@ SymbolNode {
   complexity: number
   isExported?: boolean
   typeFacts?: SymbolTypeFacts
+  duplication?: SymbolDuplicationFacts
 }
 ```
 

docs/mcp-tools.md

@@ -1,6 +1,6 @@
 # MCP Tools Reference
 
-17 tools available via MCP stdio.
+18 tools available via MCP stdio.
 
 Operation tools return JSON text payloads. Invalid operation inputs return `isError: true` with `{ "error": "..." }` using the same descriptor validation messages as CLI bad-argument exits.
 
@@ -89,7 +89,17 @@ Rank code quality and refactoring opportunities for AI agents.
 **Use when:** "What should I improve?" "Find refactoring opportunities." "Which files need tests?"
 **Not for:** Raw metric lists only (use find_hotspots or analyze_forces).
 
-## 9. get_groups
+## 9. find_duplicates
+
+Detect duplicate function families.
+
+**Input:** `{ mode?: "strict" | "mild" | "weak", minTokens?: number, skipLocal?: boolean, trace?: string }`
+**Returns:** mode, minTokens, threshold, totalCandidates, totalFamilies, families[] (id, members[], tokenCount, similarity), optional trace token evidence
+
+**Use when:** Finding copy-paste logic, renamed clones, near-miss drift, or refactor candidates.
+**Not for:** Unused code (use find_dead_exports) or module-level cohesion (use analyze_forces).
+
+## 10. get_groups
 
 Top-level directory groups with aggregate metrics.
 
@@ -99,7 +109,7 @@ Top-level directory groups with aggregate metrics.
 **Use when:** "What are the main areas of this codebase?" High-level grouping overview.
 **Not for:** Detailed module metrics (use get_module_structure).
 
-## 10. symbol_context
+## 11. symbol_context
 
 Callers, callees, and importance metrics for a function, class, or method.
 
@@ -109,7 +119,7 @@ Callers, callees, and importance metrics for a function, class, or method.
 **Use when:** "Who calls X?" "Trace this function." "What depends on this symbol?"
 **Not for:** Text search (use search) or file-level dependencies (use get_dependents).
 
-## 11. search
+## 12. search
 
 Search files and symbols by keyword.
 
@@ -119,7 +129,7 @@ Search files and symbols by keyword.
 **Use when:** "Find files related to auth." "Where is getUserById defined?"
 **Not for:** Structured call graph queries (use symbol_context).
 
-## 12. detect_changes
+## 13. detect_changes
 
 Detect changed files from git diff with risk metrics.
 
@@ -129,7 +139,7 @@ Detect changed files from git diff with risk metrics.
 **Use when:** Starting a review, triaging changes, "what changed?"
 **Not for:** Symbol-level impact (use impact_analysis).
 
-## 13. impact_analysis
+## 14. impact_analysis
 
 Symbol-level blast radius with depth-grouped risk labels.
 
@@ -139,7 +149,7 @@ Symbol-level blast radius with depth-grouped risk labels.
 **Use when:** "What breaks if I change getUserById?" Symbol-level impact assessment.
 **Not for:** File-level dependencies (use get_dependents).
 
-## 14. rename_symbol
+## 15. rename_symbol
 
 Read-only reference finder for rename planning.
 
@@ -149,7 +159,7 @@ Read-only reference finder for rename planning.
 **Use when:** Planning a rename, finding all usages of a symbol.
 **Not for:** Call graph analysis (use symbol_context).
 
-## 15. get_processes
+## 16. get_processes
 
 Trace execution flows from entry points through the call graph.
 
@@ -159,7 +169,7 @@ Trace execution flows from entry points through the call graph.
 **Use when:** "How does this app start?" "Trace request flow." "What are the entry points?"
 **Not for:** Static file dependencies (use get_dependents).
 
-## 16. get_clusters
+## 17. get_clusters
 
 Community-detected clusters of related files.
 
@@ -169,7 +179,7 @@ Community-detected clusters of related files.
 **Use when:** "What files are related?" "Find natural groupings." Discovering emergent groupings that differ from directory structure.
 **Not for:** Directory-based modules (use get_module_structure).
 
-## 17. check
+## 18. check
 
 Run the configurable rules engine and gate on findings.
 
@@ -208,6 +218,7 @@ Rules: `no-comments` (off by default), `no-circular-deps` (error), `no-dead-expo
 | "Which files need tests?" | `find_hotspots` (coverage) |
 | "What should I improve first?" | `find_opportunities` |
 | "Find refactoring opportunities." | `find_opportunities` |
+| "Where is logic duplicated?" | `find_duplicates` |
 | "What can I safely delete?" | `find_dead_exports` |
 | "How are modules organized?" | `get_module_structure` |
 | "What's architecturally wrong?" | `analyze_forces` |

llms-full.txt

@@ -31,8 +31,8 @@ Core (shared computation)
   |     typed descriptors, input schemas, CLI/MCP adapters, result wrappers, text formatters
   v
 MCP (stdio) + CLI
-  | MCP: 17 tools, 2 prompts, 3 resources for LLM agents
-  | CLI: 18 commands with formatted + JSON output for humans/CI
+  | MCP: 18 tools, 2 prompts, 3 resources for LLM agents
+  | CLI: 19 commands with formatted + JSON output for humans/CI
 ```
 
 ## Module Map
@@ -49,7 +49,9 @@ src/
   core/index.ts        <- Shared result computation (MCP + CLI)
   operations/index.ts  <- Analysis operation descriptors + typed input schemas
   operations/formatters.ts <- Result-object text formatters for CLI commands
-  mcp/index.ts         <- 17 MCP tools for LLM integration
+  parser/duplication.ts <- Function-body clone token extraction
+  duplication/index.ts <- Duplicate family detection + trace evidence
+  mcp/index.ts         <- 18 MCP tools for LLM integration
   mcp/hints.ts         <- Operation-keyed next-step hints for MCP tool responses
   impact/index.ts      <- Symbol-level impact analysis + rename planning
   search/index.ts      <- BM25 search engine
@@ -70,7 +72,7 @@ loadCodebaseGraph(rootDir)
   -> otherwise emits progress events through parse/build/analyze/cache
 
 parseCodebase(rootDir)
-  -> ParsedFile[] (with churn, complexity, test mapping)
+  -> ParsedFile[] (with churn, complexity, test mapping, symbol type facts, duplicate token facts)
 
 buildGraph(parsedFiles)
   -> BuiltGraph { graph: Graph, nodes: GraphNode[], edges: GraphEdge[] }
@@ -259,7 +261,7 @@ The most dangerous files have: high churn + high coupling + low coverage.
 
 # MCP Tools Reference
 
-17 tools available via MCP stdio.
+18 tools available via MCP stdio.
 
 ## 1. codebase_overview
 High-level summary. Input: `{ depth?: number }`. Returns: totalFiles, totalFunctions, modules, topDependedFiles, metrics, and analysis mode/call graph precision.
@@ -285,31 +287,34 @@ Unused exports. Input: `{ module?: string, limit?: number }`. Returns: files wit
 ## 8. find_opportunities
 Rank code quality and refactoring opportunities. Input: `{ limit?: number }`. Returns: ranked opportunities with priority, confidence, evidence, and suggested commands.
 
-## 9. get_groups
+## 9. find_duplicates
+Duplicate function families. Input: `{ mode?: "strict" | "mild" | "weak", minTokens?: number, skipLocal?: boolean, trace?: string }`. Returns: family IDs, members, token counts, similarity thresholds, and optional trace evidence.
+
+## 10. get_groups
 Top-level directory groups. Input: `{}`. Returns: groups with rank, files, loc, importance, coupling.
 
-## 10. symbol_context
+## 11. symbol_context
 Function/class/method context. Input: `{ name: string }`. Returns: callers, callees, metrics, additive typeFacts when known.
 
-## 11. search
+## 12. search
 Keyword/search shape facts (BM25). Input: `{ query: string, limit?: number }`. Returns: ranked files + symbols with additive typeFacts when known.
 
-## 12. detect_changes
+## 13. detect_changes
 Git diff analysis. Input: `{ scope?: "staged" | "unstaged" | "all" }`. Returns: changed files, affected files, risk metrics.
 
-## 13. impact_analysis
+## 14. impact_analysis
 Symbol-level blast radius. Input: `{ symbol: string }`. Returns: depth-grouped impact levels.
 
-## 14. rename_symbol
+## 15. rename_symbol
 Reference finder for rename planning. Input: `{ oldName: string, newName: string, dryRun?: boolean }`. Returns: references with confidence.
 
-## 15. get_processes
+## 16. get_processes
 Entry point execution flows. Input: `{ entryPoint?: string, limit?: number }`. Returns: processes with steps and depth.
 
-## 16. get_clusters
+## 17. get_clusters
 Community-detected file clusters. Input: `{ minFiles?: number }`. Returns: clusters with cohesion.
 
-## 17. check
+## 18. check
 Rules-engine gate. Input: `{}`. Returns: pass/warn/fail verdict, findings, config path, and summary counts.
 
 ## Tool Selection Guide
@@ -324,6 +329,7 @@ Rules-engine gate. Input: `{}`. Returns: pass/warn/fail verdict, findings, confi
 | Which files need tests? | find_hotspots (coverage) |
 | What should I improve first? | find_opportunities |
 | Find refactoring opportunities | find_opportunities |
+| Where is logic duplicated? | find_duplicates |
 | What can I safely delete? | find_dead_exports |
 | How are modules organized? | get_module_structure |
 | What's architecturally wrong? | analyze_forces |
@@ -339,7 +345,7 @@ Rules-engine gate. Input: `{}`. Returns: pass/warn/fail verdict, findings, confi
 
 # CLI Reference
 
-18 commands — 16 analysis commands (full parity with MCP tools), `check` for CI rules, and `init` for agent adoption.
+19 commands — 17 analysis commands (full parity with MCP tools), `check` for CI rules, and `init` for agent adoption.
 
 ## Commands
 
@@ -403,6 +409,12 @@ codebase-intelligence opportunities <path> [--limit <n>] [--json] [--force]
 ```
 Rank code quality and refactoring opportunities with evidence, confidence, and suggested commands.
 
+### duplicates
+```bash
+codebase-intelligence duplicates <path> [--mode <mode>] [--min-tokens <n>] [--skip-local] [--trace <id>] [--json] [--force]
+```
+Detect strict, renamed, and near-miss duplicate function families.
+
 ### groups
 ```bash
 codebase-intelligence groups <path> [--json] [--force]