feat: add llm:compile-error for validating LLM-generated NetLogo code - #69
Open
JNK234 wants to merge 4 commits into
Open
feat: add llm:compile-error for validating LLM-generated NetLogo code#69JNK234 wants to merge 4 commits into
JNK234 wants to merge 4 commits into
Conversation
Reports "" if a string of NetLogo commands compiles, otherwise the compiler's error message and character offset. Calls NetLogo's own compiler via workspace.compileCommands in Turtle context, so a non-empty result guarantees `run` would have failed with the same error. The live model's symbol table is in scope, which means turtles-own variables, globals and breeds all resolve — something an external validator cannot do, and the source of the most common failure class in LLM-generated agent rules. Scoped deliberately to syntax only. Domain policy (banned primitives, length limits, required commands) stays in NetLogo where it can change without rebuilding the extension. Tests cover valid code, invalid code, turtle context, and a regression corpus of 14 real evolved rules from the LEAR project. That corpus guards against false negatives: rejecting a valid rule is the unacceptable failure for an evolutionary system. It includes `rt (random 90) * 0.5 + (random 45) * 0.5`, which LEAR's hand-written Python validator rejects as "too complex for current parser". Refs #52
llm:compile-error code
(llm:compile-error code ["die" "clear-all"])
Extends the existing primitive rather than adding a second one: both
checks answer the same question — is this code acceptable to run — and
share the same "" / message contract, so callers write one check either
way. Uses defaultOption, the same variadic pattern as the csv extension.
Syntax is checked first. Code that does not compile cannot run, so a
banned-primitive report would be noise, and tokenizing malformed code is
unreliable.
Matching uses NetLogo's own tokenizer (org.nlogo.lex.Tokenizer) on exact
tokens, not substrings. Tests cover the three false positives substring
matching would produce: `die` in a comment, a variable named `diehard`,
and the string literal "die". Rejecting valid code is the failure mode
that matters here, so those cases are locked in.
The list is caller-supplied; the extension ships no built-in set, because
what counts as dangerous depends on the model.
Refs #52
…ned list Adds 8 test blocks covering behaviour that was previously unverified: - multi-line code with `let` locals, including scoping violations (use-before-define, redefinition in the same scope) - turtle-own and built-in variables resolving against the live model - comments, blank lines and indentation - nested blocks, `repeat`, `foreach`, multi-branch `(ifelse ...)`, unbalanced brackets - banned primitives across multiple lines and inside nested blocks - multiple banned primitives found at once (reported in source order) - whitespace and empty entries in the banned list Fixes a real gap found while writing them: a banned list containing non-strings, e.g. `[1 2]`, silently skipped those entries and returned "" — a false all-clear for a caller who passed the wrong thing. It now throws an ExtensionException naming the offending values. Exact compiler messages are asserted where they are stable, so a change in NetLogo's wording surfaces as a test failure rather than drifting silently through the docs. 54 tests passing. Refs #52
chat-with-template appeared only as a row in the quick-reference table —
no section, no example, no description of the YAML format. Users had to
read demos/templates/ or the Scala source to learn the file layout.
Documents the actual behaviour, verified against the implementation:
- template: required, system: optional
- file lookup order is model directory, then literal path, then working
directory; keeping the template beside the .nlogox is the reliable form
- an unmatched {placeholder} is left as literal text rather than raising
- the system message applies to the call only and is not stored in history
- rendered prompt and response are committed to history together on
success, and nothing is committed on failure
Worked example uses a foraging decision, which is the pattern the
templates in demos/ are actually for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #52 (roadmap B3).
Why
Several demos use the "LLM writes NetLogo code, the model runs it" pattern (
demos/templates/code-evolution-template.yaml,movement-evolution.yaml). Today generated code goes straight torun/runresultwith no checks: invalid code throws mid-tick, and nothing stops a model from emittingclear-allordie.This adds the gate that makes that pattern safe enough to put in front of students.
What
A single reporter,
llm:compile-error, returning""when code is acceptable and an explanatory message otherwise.Design decisions
Uses NetLogo's own compiler, reached via
ExtensionContext→workspace.compileCommands. This answers the issue's first open question: an extension can compile arbitrary source without a full workspace round-trip. It also means the live model's symbol table is in scope, soturtles-ownvariables, globals, and breeds resolve — an external validator cannot do this, and undefined-variable errors are the most common failure caught.Compiles in turtle context, because generated agent rules are normally executed inside
ask turtles. Observer context would rejectfd/rtand refuse valid rules.Syntax is checked first and returns early. Code that does not compile cannot run, so reporting banned primitives on it would be noise — and tokenizing malformed code is unreliable.
Banned-primitive matching is on exact tokens via NetLogo's tokenizer, not substrings. Banning
dierejectsdiebut notdiehard, notdiein a comment, and not the string"die". Substring matching produced all three false positives, and a false positive discards code that was valid.No built-in banned set ships. What counts as dangerous is model-specific — a model may legitimately need
hatchwhile forbidding it in generated rules. The issue suggested a default list; this was deliberately not taken, so policy stays in NetLogo where it can change without rebuilding the extension.A malformed list is rejected rather than ignored.
(llm:compile-error "die" [1 2])previously returned""— a false all-clear for a caller who passed the wrong thing. It now raises.Scope
This proves code compiles, not that it cannot throw at runtime. Bounds errors such as
item 3on a three-element list still surface only during execution, socarefullyaroundrunis still needed. Documented in the API reference.Tests
sbt test— 54 passing, 0 failed. 15 new blocks intests.txt(~70 assertions):letscoping, comments, nested blocksFollow-ups, not in this PR