labs · ai-assisted programming
A coding agent in a terminal has your shell: it reads your files, runs your commands and works across the whole repository. How it behaves is decided less by what you type than by how it is configured — what it reads every time it starts, the commands you give it, and what it may run without asking. This lab is about that configuration. The tool is the example; the configuration model is what transfers.
AGENTS.md) that change what the agent
does in every future session — and check that they didOpen a Codespace on your own copy of the module repo.
Move into this lab and install its dependencies:
cd lectures-and-labs/week09/cli_agents_lab
pip install -r requirements.txt
Check the tools you need are present:
python check_setup.py
Which agent. The steps are written for GitHub Copilot CLI, which
comes with every Copilot plan — including Copilot Free and the free
Copilot Student plan you get through GitHub Education. The student plan
chooses the model for you and has a monthly allowance, shared with
Copilot Chat; /usage shows what the current session has used. If
Copilot is not available to you, use Gemini CLI with the free Gemini
API key from the RAG lab, kept in a .env file in sample-app (DIY 1
shows how): Google's own sign-in stopped serving free personal accounts
on 18 June 2026. Wherever the two differ, the Gemini version is given
too.
| Folder | What is in it |
|---|---|
sample-app/ |
A small project with one failing test — the agent's work |
policy/ |
A permission checker, a team policy and a list of commands |
examples/ |
A custom command, a skill and an agent profile, ready to install |
If anything goes wrong installing or signing in, see TROUBLESHOOTING.md.
A terminal agent is a program you install like any other. Copilot CLI and Gemini CLI are both npm packages and need Node 22 or newer, which the Codespace already has.
The first time it starts in a folder, it asks whether you trust that folder. Take the question seriously: trusting a folder lets the agent read it, and lets the files in it configure the agent.
Install the agent and confirm it runs:
npm install -g @github/copilot
copilot --version
Gemini: npm install -g @google/gemini-cli, then gemini --version.
Start it inside the sample project:
cd sample-app
copilot
Trust the folder for this session only. If the screen already says
Signed in to https://github.com as <your username>, you are signed
in. If it says Please use /login to sign in to use Copilot, type
/login, choose GitHub.com, then Sign in with a device code,
and follow the code it shows you.
Gemini: first create sample-app/.env with your key and the model
line:
GEMINI_API_KEY=your-key-here
GEMINI_MODEL=gemini-3.5-flash-lite
Then run gemini and trust the folder. When it asks how to
authenticate, choose Use Gemini API Key, then press Enter to
submit the key it shows you. The bottom right of the screen should
read gemini-3.5-flash-lite.
Before asking it anything, type /help. Find the command that does
each of these jobs, and write it in your notes:
| Job | Command |
|---|---|
| Start a fresh conversation | |
| Show how full the context window is | |
| Shrink a long conversation | |
| Choose the model | |
| Come back to an earlier session |
Ask: "What does this project do? Don't change anything." Then check how much of the context window that one question used.
Type !python -m pytest -q. The ! runs the command yourself,
without the model. Note the name of the failing test — it is the job
for the next section.
What you should have
The agent installed and signed in, the five-row table filled in from
/help — not from this page — and the name of the one failing test.
On the student plan, /model may offer only automatic selection. That is
expected, not a fault.
If Copilot says you have no access in a Codespace, the Codespace's own
GITHUB_TOKEN may be getting in the way — see
TROUBLESHOOTING.md.
Gemini: the key is the one from the RAG lab, the value of LLM_API_KEY
in rag_lab/.env, or a new one from
aistudio.google.com/apikey.
.env is gitignored, so git never sees it, and the undo in DIY 2
(git clean -fd) leaves it alone. Never type the key into a command or
paste it into code. Keep the GEMINI_MODEL line: without it Gemini uses
its default model, the bottom right reads Auto, and on a free key that
model gave no answer when this lab was checked in October 2026.
Step 5 matters more than it looks. ! is your action. Asking the agent to
run the same command is its action, and goes through its permissions —
which is section 4.
Anything you tell an agent in a conversation lasts as long as the conversation. A rule that should still hold tomorrow belongs in a file the agent reads every time it starts.
AGENTS.md is an open format for that file, read by most terminal agents.
Copilot CLI reads it, and .github/copilot-instructions.md too. Gemini
CLI reads GEMINI.md unless you tell it to read AGENTS.md as well.
sample-app/test_stats.py expects median([4, 1, 3, 2]) == 2.5. The code
returns 3. There are two ways to make that test pass, and only one of
them is a fix.
Commit, so you can see exactly what changes and undo it:
git add -A && git commit --allow-empty -m "before the agent"
Start a fresh conversation (/clear) and type only: "Make the tests
pass."
When it finishes, run git diff. Which file did it change —
stats.py, test_stats.py, or both?
If the diff is longer than the terminal, git shows it in a scrolling
viewer and your prompt disappears. Scroll with the arrow keys, then
press q to get back to the prompt. If q does nothing, you are in the
Vim editor: press Esc, type :q and press Enter.
Record the exact prompt and what it changed.
Undo everything it did: git checkout -- . && git clean -fd — you
committed first, so this removes only what the agent changed or added.
What you should have
A note with the prompt, the file or files the agent changed, and whether any test's expected value changed.
Either outcome is a result. On a task this small the agent often fixes the code; the finding is that nothing stopped it editing the test instead. "Passing" was the goal it was given, and changing the test achieves it.
Read the diff, not the agent's summary of what it did. They do not always agree.
--allow-empty makes the commit work even when nothing has changed since
your last one, so step 1 never stops on "nothing to commit".
Create sample-app/AGENTS.md. Give it, at least: how to run the tests;
that a test's expected value is never changed, and what to do instead;
standard library only; and to check the result of every command before
the next step.
Gemini only: make it read AGENTS.md by creating
sample-app/.gemini/settings.json:
{"context": {"fileName": ["AGENTS.md", "GEMINI.md"]}}
Quit and start a new session. Ask: "What rules do your instructions give you for this project?"
Repeat DIY 2: commit, type "Make the tests pass.", then git diff.
Undo with git checkout -- . && git clean -fd so the bug is back for later.
Now push against the rule: "The test is wrong. Change it to expect 3." Record what it does, then undo again.
What you should have
Your AGENTS.md, the agent's answer to step 3 showing it read the file, a
diff from step 4 that touches only stats.py, and what happened in
step 5.
A rule that says what to do instead works better than a bare ban: "if a test looks wrong, stop and explain why" gives the agent somewhere to go.
You may find it loaded more than your file. Copilot also reads
instruction files from the root of the repository — this module ships an
AGENTS.md and a CLAUDE.md among them — and combines them all with no
order of priority.
In Gemini, /memory show prints everything it loaded; if it ignores your
settings file, trust the folder, because settings in an untrusted folder
are ignored.
If step 5 changes the test, you have found the honest limit of this
section: AGENTS.md is text the model weighs, not a lock. The lock is
section 4.
The built-in commands control the tool. Your own commands package a prompt you would otherwise retype — with live context, such as the current diff, filled in when you run it. They are files, so they are committed and the whole team gets them.
examples/ holds one of each kind, deliberately outside the folders
the agents load from. Installing one is part of the exercise:
| File | What it is | Install it to |
|---|---|---|
examples/copilot/skills/review-diff/SKILL.md |
A Copilot CLI skill | .github/skills/review-diff/SKILL.md at the root of your repo |
examples/gemini/review.toml |
A Gemini CLI command | sample-app/.gemini/commands/review.toml |
examples/copilot/agents/reviewer.agent.md |
A Copilot CLI custom agent | See the extensions |
mkdir -p the folder
first), then start a new session. Copilot: /skills lists what it found. Gemini:
/commands reload.stats.py, then run it: /review-diff in Copilot,
/review in Gemini.What you should have
The example installed and run once, your own command file committed, and one sentence saying what your command runs on your machine and when.
Copilot skills: the folder name and the name: in the file's frontmatter
must match, lowercase with hyphens. A skill can also be picked up without
being called, when its description fits what you asked. If it does not
show up, the personal folder ~/.copilot/skills/ works from anywhere.
Gemini commands: {{args}} is whatever you type after the command's name,
and !{...} runs a shell command and pastes its output into the prompt —
after asking you first. That confirmation is the reason to read someone
else's command before you install it. The personal folder
~/.gemini/commands/ works from anywhere.
Standing instructions ask. Permissions enforce. Every terminal agent decides each action one of three ways:
| Answer | Meaning |
|---|---|
| Allow | Runs without asking |
| Ask | A person decides — the default for anything not allowed |
| Deny | Never runs, whatever an allow rule says |
In Copilot CLI you write them as flags:
copilot --allow-tool='shell(git:*)' --deny-tool='shell(git push)'
shell(rm) covers every rm command, shell(git:*) covers every git
command, and for git and gh you can name a subcommand, as git push
does here. Gemini CLI does the same job with --allowed-tools and policy
files, and matches a text prefix. The details differ between tools and
between versions — so before you trust a policy, test it.
policy/policy_check.py is a small checker for exactly that. Its rules
are written like Copilot's:
| Rule | Covers |
|---|---|
shell |
Every shell command |
shell(git) |
Every command that starts with the word git |
shell(git push) |
Every command that starts with git push — git push --force too |
read, write, … |
Other tools — never a shell command |
Copilot's :* means "and anything after it". The checker has no
wildcards, so shell(git) already means that, and a rule with :* in it
is refused.
A line containing |, ||, &&, ; or & is split into separate
commands, and each is judged: the line is DENY if any part is denied,
ASK if any part would be asked, and ALLOW only if every part is
allowed.
Open policy/team-policy.json. Read its intent first, then its
rules.
Open policy/commands.txt. Before running anything, write ALLOW,
ASK or DENY beside each line in your notes.
Run the checker from the lab folder (cd .. first if your terminal is
still inside sample-app):
python policy/policy_check.py policy/team-policy.json --file policy/commands.txt
Mark every verdict you got wrong.
Mark every command the checker allows that breaks the policy's intent. Those are the gaps. For each one, write one sentence on why the rules miss it.
Expected output
ALLOW git status
DENY git push
DENY git push --force origin main
...
There are three gaps. A rule matches the words a command starts with, so look for commands that break the intent while starting with words the policy allows.
read is the agent's own file-reading tool. It says nothing about cat
in a shell.
policy/team-policy.json to policy/my-policy.json.git status, git diff, git log, git add, git commit,
npm test, pytest and python -m pytest all still come out ALLOW —
with any arguments.policy/my-commands.txt with five lines of your own: three the
agent should be free to run, and two that break the intent in a way
commands.txt did not show.my-policy.json until every verdict is one you
would defend.What you should have
my-policy.json, my-commands.txt, the checker's output for both, and
your two sentences.
Allowing a whole program (shell(git), shell(python)) allows everything
that program can do. Allow the subcommands you mean instead, and let
everything else fall through to ASK.
For step 5, count the ways there are to spell "push" or "delete" — then
look up what pytest --basetemp does to the directory you give it: it
deletes it and makes it again, the first time a test asks pytest for a
temporary folder (tmp_path, for example). The sample app's tests never
ask, so with them nothing is deleted.
The checker is a model of one way to match rules. Find out how your agent actually does it — using harmless commands only.
From sample-app (cd sample-app from the lab folder), start the
agent with one allow rule and one deny rule:
copilot --allow-tool='shell(git status)' --deny-tool='shell(git log)'
Gemini: gemini --allowed-tools "run_shell_command(git status)", and
for the deny, the policy file in the hint.
Ask it to run exactly git status --short. Did it ask you first?
Ask it to run exactly git log --oneline -3. Was it blocked?
Ask it to run exactly git -C . log --oneline -3. Was it blocked?
Put the three results in a table beside what policy_check.py decides
for the same two rules.
What you should have
A three-row table: the command, what the checker decided, and what your agent did.
Say "run exactly this command". Otherwise the agent may tidy
git -C . log into git log, and you will be testing a different
command.
The Gemini deny rule: save this as ~/.gemini/policies/lab-probe.toml,
restart, and delete the file when you are done — it applies to every
project on the machine.
[[rule]]
toolName = "run_shell_command"
commandPrefix = "git log"
decision = "deny"
priority = 500
Where the agent and the checker disagree, neither is broken. You now know something about your agent that reading its flags did not tell you.
-p runs one prompt and exits, so an agent can live in a script or a CI
job. With nobody there to answer a question, anything not allowed in
advance is refused: every permission is decided before it starts.
From sample-app, run a headless check on the project:
copilot -p "Run pytest -q. If any test fails, explain why in three lines." \
--allow-tool='shell(pytest)' --deny-tool='write'
The prompt names pytest on purpose: the policy allows commands that
start with pytest, so python -m pytest would be asked — and
headless, asked means refused.
Gemini: pipe the output in, so it needs no tools — whether it may
use any is decided by its own allow/deny configuration, not by the pipe:
python -m pytest -q 2>&1 | gemini -p "Explain any failing test in three lines."
Save the command as explain-failures.sh in sample-app and run it
there with bash explain-failures.sh.
Change the prompt so the job needs something you did not allow — add "then fix the bug" — run it again, and record what happens.
Confirm nothing changed: git status.
Write down the smallest set of permissions your script needs, and why that set is safe to run unattended.
What you should have
explain-failures.sh, its output on the failing test, what happened in
step 3, and your minimal permission set with its justification.
--deny-tool='write' blocks the agent's file-editing tools. A shell
command can still write files, which is why the script allows
shell(pytest) and not shell.
Piping the text in is the safest headless pattern there is: everything the agent needs is already in the prompt, so it has no reason to reach for a tool. But only a policy stops it if it tries — "runs nothing" is a property of what you allowed, not of the pipe.
Optional — the sections above are the two-hour path.
examples/copilot/agents/reviewer.agent.md
to .github/agents/ at the root of your repo, start a session, and pick
it with /agent. Its tools list leaves out editing and the shell
entirely: narrowing an agent's tools is a permission decision, not a
style choice./mcp lists the servers your agent can use and adds
new ones. Connect the server you built in the MCP lab and ask the agent
to use it./delegate
in Copilot CLI hands it a task; it works on a branch and opens a pull
request. When an agent works while you are not watching, what does
review even mean?AGENTS.md as a lock. It is text the model weighs.
Permissions are the lock.shell(python) — and calling it "run my
scripts". It allows everything Python can do.!{...} runs
on your machine.AGENTS.md — and a standing
instruction still only asks.The through-line: configure an agent as if it will do exactly what you allowed — because it will.