Skip to content

Make the system prompt configurable via .codereview.yml (like rules) #2

Description

@hazardemircan

Summary

Today, rules: in .codereview.yml lets users inject project-specific review
guidance, but the rest of the system prompt — the reviewer persona and the
severity definitions — is hardcoded in internal/ai/review.go.
Teams reviewing very different codebases (a fintech backend vs. a frontend, say)
want to tune that wording without forking the tool.

This issue adds a systemPrompt: field to .codereview.yml that overrides the
reviewer guidance (persona + severity definitions), while the tool keeps
enforcing the machine contract (the JSON output schema) in code so a custom
prompt can never break response parsing.

Background: how the prompt is built today

Reviewer.systemPrompt() (in internal/ai/review.go) concatenates four parts:

  1. Persona — "You are an expert code reviewer integrated into a CI/CD pipeline."
  2. Output contract — the JSON findings schema, field names/types, and that
    severity must be one of blocker|major|minor|info.
  3. Severity definitions — what each severity means.
  4. Project rules — from cfg.Rules.

Output contract part is load-bearing: the response is parsed as JSON in ReviewFile
(json.Unmarshal(...Message.Content...)). It must stay fixed.

Proposed design

Split the prompt into a configurable guidance block (default = persona +
severity definitions) and a fixed contract block (always appended in code),
then append rules as today. Assembled order:

<systemPrompt guidance>     # configurable; default lives in code
<fixed JSON output contract># always appended, never user-editable
<project rules>             # appended from cfg.Rules, unchanged

When systemPrompt is absent, behavior must be identical to today (the
default value reproduces the current persona + severity wording).

Config example

# .codereview.yml
systemPrompt: |
  You are a senior Go reviewer for a fintech team.
  Be strict about concurrency and money math.

  Severity guide:
    blocker: anything that could produce an incorrect balance or lose money
    major:   missing error handling, race conditions
    minor:   style, naming
    info:    suggestions

# rules: still append after the system prompt, as before
rules:
  - "All monetary values must use the Money type, never float64"

The tool always appends (not user-editable): the JSON findings schema, the
allowed severity values, the "only flag added lines / return {"findings":[]}
if none" rule, and then the rules: list.

Implementation guide

  1. internal/config/config.go

    • Add SystemPrompt string yaml:"systemPrompt"`` to Config.
    • Add an exported DefaultSystemPrompt (constant or func) holding the current
      persona + severity-definitions text.
    • In applyDefaults(), set SystemPrompt = DefaultSystemPrompt when empty.
  2. internal/ai/review.go

    • Pass the guidance into the reviewer (add a param to NewReviewer, or pass
      the whole *config.Config). Store it on Reviewer.
    • Rewrite systemPrompt() to emit: guidance + the fixed contract block +
      the rules. Move the JSON-schema/output-rules text into the fixed block so it
      is never overridable. (Rename the method or field to avoid a name clash.)
  3. main.go

    • Update the ai.NewReviewer(...) call to pass cfg.SystemPrompt.
  4. Docs

    • README: document systemPrompt in the Configuration section, show the
      default, and state clearly that the JSON output contract is always enforced
      regardless of the custom prompt.
    • .codereview.yml: add a commented systemPrompt example.
  5. Tests

    • config_test.go: absent → default applied; present → custom value kept.
    • review_test.go: the built prompt contains the custom guidance and
      always contains the JSON-contract text and the rules.

Acceptance criteria

  • systemPrompt in .codereview.yml changes the persona/severity guidance
    sent to the model.
  • Omitting systemPrompt produces the same review behavior as before.
  • The JSON output contract and severity value list are always present in
    the prompt, even with a custom systemPrompt, and responses still parse.
  • rules: still works and is appended after the system prompt.
  • README and .codereview.yml document the new field and the enforced contract.
  • go test ./... passes with new coverage for both config and prompt assembly.

Out of scope (possible follow-ups)

  • An env-var override (SYSTEM_PROMPT) — rules is config-only, so keep parity.
  • Loading the prompt from a separate file (systemPromptFile:) for very long prompts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions