execpolicy: Let the Policy File Carry Its Own Tests
Codex writes positive and negative examples into the rule itself: how a command is split, prefix-matched, and judged at the strictest level, and how load-time tests pin down false hits
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “execpolicy: Let the Policy File Carry Its Own Tests”?
Codex writes positive and negative examples into the rule itself: how a command is split, prefix-matched, and judged at the strictest level, and how load-time tests pin down false hits
Follow the handoffs, not the demo. A system becomes dependable at the boundaries between model, tools, state, permissions, and people. Read each handoff as a place where you can observe, test, and recover.
Name the input, owner, approval, and recovery action for one automated step.
A successful run that cannot explain what happened or be safely repeated.
argv this step sees
Tests the policy carries
- Read the policy text and start parseparser.rs L57
- Starlark evaluates prefix_rule; missing decision defaults to allowparser.rs L348
- Rules and examples go onto the pending-check queueparser.rs L405
- Run negatives first; a hit reports ExampleDidMatchparser.rs L145
- Then positives; a miss reports ExampleDidNotMatchparser.rs L147
- Wrapped commands are split into inner argv firstexec_policy.rs L831
- Exact-lookup the rule bucket by the first tokenpolicy.rs L334
- Multiple rules or a compound command take the strictestpolicy.rs L403
- The call maps to Skip, NeedsApproval, or Forbiddenexec_policy.rs L375
You write a ban for the team, pattern as git plus reset. You meant to stop --hard. A week later someone reports --keep is blocked too. Shorten the prefix and anything after it hits the same rule.
Another week on, the model wraps it in bash -lc. The ban matches literal argv against bash, the first token misses, and the command slides into the heuristic path.
Every whitelist line is betting two things. One, the pattern covers what you meant to stop. Two, it won’t false-hit what you meant to let through. You usually only learn that in production. The next person who edits the pattern also can’t see which command the author was afraid of clipping.
Codex writes the examples into the rule itself. match is the command this rule must hit; not_match is the one it must miss. prefix_rule doesn’t run examples on the spot — it queues the rule and the examples. After the whole Starlark file evaluates, parse checks them together.
Order is fixed. Negatives first, then positives. A false hit on a negative reports ExampleDidMatch. If no positive hits, it reports ExampleDidNotMatch. Either error makes parse fail, and Policy is never builded out.Source: codex-rs/execpolicy/src/parser.rs lines 57–78 and 133–151
The crate README puts it in one line: they are sample invocations checked at load time, and you can treat them as unit tests. Both the string form and the token-array form become a token list in the parser; strings go through shlex. During the check the heuristic callback is empty, so a positive must hit a real prefix rule — it can’t ride a heuristic.
The typical whitelist failure is the author thinking the pattern means one thing. Write that belief down as examples and the machine can disagree. This doesn’t depend on Starlark — copy it into JSON or YAML and it still holds.
Keep the rule and the examples together, and when the policy file is copied into a user directory or shipped as an overlay, the examples travel with it. The loader has no branch that reads rules and skips examples. Skipping examples is skipping tests. The loader won’t invent examples for you; if you wrote them, it will enforce them.
A regex can say git reset plus anything, and it can also write exceptions the author can no longer read. If stacked rules take the widest, one loose user-level allow can cover a system-level forbidden.
The rule body is a prefix. Matching is exact string equality — no glob, no regex. git reset --hard will eat a command that then adds origin/main, because extra tokens don’t join the compare. It will not eat a --config stuffed in the middle, because the second token misses.Source: codex-rs/execpolicy/src/rule.rs lines 46–59
Rules go into buckets by the first token. Lookup exact-matches argv[0] first. If that misses, it may fold an absolute path down to a basename. On multiple hits it takes max of decision. Decision derives Ord; variant write order is severity: Allow < Prompt < Forbidden.
Compound commands are split, flattened, then maxed again. One segment git status is Prompt, one git commit is Forbidden, and the whole pipe is still Forbidden.Source: codex-rs/execpolicy/src/policy.rs lines 265–287 and 402–411
Stacking can only tighten, never loosen. That order lives on the enum variants; runtime has no second priority table to get wrong. Rewrite it in another language: three strings as an ordered enum, one max to fold them.
A prefix forces you to land intent as a token sequence. The cost: a flag stuffed in the middle kills the rule, so the author must shorten the prefix or write another. The examples are there to catch that cost: write it too short and the negative dies at load time.
The ban is written as git reset --hard. The model wraps it in bash -lc. Match literal argv and the first token is bash — the ban stays dark.
Before judging, it runs parse_shell_lc_plain_commands. The script may only be plain-word commands plus &&, ||, semicolons, and pipes — no redirects, substitutions, parens, or control flow. Once it passes, command nodes become argv segments. bash -lc wrapping git reset --hard splits into those three inner tokens, then those go to prefix match.Source: codex-rs/core/src/exec_policy.rs lines 831–858
If it won’t split, the whole argv is one command, handed to heuristics and the later sandbox. Empty quotes in an argument can still be recovered. Empty quotes in the command name fail word-only parse, the prefix rule never sees git, and the command drops into heuristics.
The parser admits what it can’t swallow, so it doesn’t pretend it already understood the script. That’s the generic fail-closed shape. It stops a successful parse that dropped a dangerous segment. After a failed split, heuristics and the sandbox are still there.
DeepSeek Harness: two knobs and a dropdown
DSH doesn’t write command-level patterns. Permission presets bundle sandbox mode and approval policy. The default table has two rows: workspace-write with ask, danger-full-access with never. Approval policy itself is only ask and never.
The knobs are easy. You cannot write a preset that only bans git reset --hard and leaves other git alone. That exception either goes to the escalate-after-sandbox-deny path, or to a custom knob combo shown under the reserved name custom. The dropdown covers everyday switches; the long tail of named bans it cannot cover.
Claude Code: tool name plus an optional-content allowlist
A rule string looks like Bash, or Bash(npm install), or Bash(git *). The parser splits tool name and content on the parens. Shell rules then split into exact, prefix, and glob.
Search match, not_match, example as rule fields: there is no load-time example check. Once git * is in allow, git reset --hard is eaten by that glob unless you write a more specific deny. Codex pins exceptions at load time with a shorter exact prefix plus a negative. Claude Code leaves exceptions to rule stacking order and a runtime confirm.
Prefix too short: load or judge?
Shorten the ban on git reset --hard to git plus reset, and leave not_match as --keep. At load, do you get ExampleDidMatch or ExampleDidNotMatch, and does the command still get a chance to be judged?
A harder follow-up: under that same bad rule, does ls -l still get allow first? In the demo, switch the policy to “Prefix too short” and play it again — check it against your reasoning.
The handoffs inside “Try it first · One command through the policy”
“You write a ban for the team, pattern as git plus reset.” shows that an Agent is not defined by the model alone. Each handoff between model, context, tools, state, permissions, and people affects both progress and recovery.
Write the state before adding capability
Starting from “Another week on, the model wraps it in bash -lc .”, split the workflow into starting state, next action, tool result, state update, and stop condition. Debugging then means finding the first lost piece of information or authority instead of saying vaguely that the model “got worse”.
- Read the policy text and start parse parser.rs L57
- Starlark evaluates prefix_rule; missing decision defaults to allow parser.rs L348
- Rules and examples go onto the pending-check queue parser.rs L405
A happy path is not reliability
Use “A harder follow-up: under that same bad rule, does ls -l still get allow first?” to replay one successful and one failed run. Record the context, tool result, and owner at each turn; the workflow is maintainable when a second person can follow it without the original builder.
From “Try it first · One command through the policy” to “Idea 1 · Keep the rule and the examples in one place”
“Try it first · One command through the policy” grounds the problem in “One policy, four commands, three writings Play Step Reset Cmd ls -l git reset --hard git reset --keep bash -lc wrap Safe, dangerous-flagged, looks dangerous, wrapped as a disguise. Pick one, then play. Policy C…”. “Idea 1 · Keep the rule and the examples in one place” then moves it toward “You write a ban for the team, pattern as git plus reset. You meant to stop --hard . A week later someone reports --keep is blocked too. Shorten the prefix and anything after it hits the same rule”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
When analyzing an Agent, trace state, action, tool result, and next step in order. Each handoff should explain where information came from, who confirmed it, and where failure stops.
- “Try it first · One command through the policy”: One policy, four commands, three writings Play Step Reset Cmd ls -l git reset --hard git reset --keep bash -lc wrap Safe, dangerous-flagged, looks dangerous, wrapped as a disguise. Pick one, then play. Policy C…
- “Idea 1 · Keep the rule and the examples in one place”: You write a ban for the team, pattern as git plus reset. You meant to stop --hard . A week later someone reports --keep is blocked too. Shorten the prefix and anything after it hits the same rule
- “The closing point”: Then positives; a miss reports ExampleDidNotMatch parser.rs L147
The final “The closing point” brings the discussion to “Then positives; a miss reports ExampleDidNotMatch parser.rs L147”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.