labs · ai-assisted programming

AIAP Automated Checking Lab

Two halves. Build a pipeline that checks code which behaves the same every time, then build one that checks something that answers differently on every run — because any app with an AI feature in it contains both.

What you'll learn

• Build a GitHub Actions pipeline from nothing and read its output
• Order checks by irreversibility rather than by sophistication
• Put an AI review step in a pipeline without letting it lie to you
• Write an eval suite for a non-deterministic feature
• Use the assertion ladder, and pick the strongest rung a task allows

Table of Contents

1. A pipeline from nothing
2. The checks that matter most
3. AI inside the pipeline
4. Evals: checking the unrepeatable
5. Common mistakes
6. Summary

Getting started

1. Open a Codespace on your own copy of the module repo.

2. Move into this lab and install its dependencies:

   cd lectures-and-labs/week10/cicd_lab
   pip install -r requirements.txt
   

3. Confirm the sample app's test passes, then run the app and open the port the Codespace forwards:

   python -m pytest
   python -m flask --app startup run
   

hello_app/ is a small Flask application with a working test. It is deliberately ordinary — the lab is about what you build around it.

Section 3 sends a diff to a hosted model from GitHub Actions. It uses the same free API key you got for the RAG lab, stored in your copy of the repo as a repository secret named LLM_API_KEY — nothing in this lab costs money, but on the free tier Google may use what you send to improve its products, so send it nothing personal. Sections 1, 2 and 4 need no key at all.

1. A pipeline from nothing

A pipeline is a script that runs on a trigger. The only thing that makes it different from running the commands yourself is that it runs whether or not you remembered.

DIY 1: Make it run your tests

1. Create .github/workflows/ci.yml in your own copy of the repo.
2. Trigger it on push.
3. Have it check out the code, set up Python 3.12, install lectures-and-labs/week10/cicd_lab/requirements.txt, and run python -m pytest from lectures-and-labs/week10/cicd_lab.
4. Push, and watch it in the Actions tab.
5. Now break a test on purpose, push, and confirm the run goes red.

Expected output

Run python -m pytest
========================= test session starts =========================
collected 1 item

tests/test_app.py .                                             [100%]

========================== 1 passed in 0.4s ===========================

2. The checks that matter most

Most people add tests first because tests are what CI is "for". Value per minute says otherwise.

DIY 2: Add the cheap checks first

Add three checks to your pipeline, in this order:

1. Secret scanning — GitHub's own scanner is for public repos and paid plans, and your copy is private, so add gitleaks to ci.yml instead. Give the checkout step fetch-depth: 0 and add the gitleaks step:

         - uses: actions/checkout@v7
           with:
             fetch-depth: 0
         - uses: gitleaks/gitleaks-action@v3
   

2. Dependency scanning — enable Dependabot alerts (Settings → Advanced Security).

3. A linter — add a ruff or pylint step to ci.yml.

4. For each, record in findings.md: what it caught, or that it caught nothing.

5. Answer: which of these three could you not undo by pushing a fix?

What you should have

Three checks in place — gitleaks and a linter in ci.yml, Dependabot alerts on — and a written answer to step 5 with your reasoning.

3. AI inside the pipeline

A linter finds style. A compiler finds type errors. Neither can tell you that a function's name no longer describes what it does.

review_diff.py in this folder is the starter for this section. It reads a diff, sends it to a model with a prompt that asks only for the things a compiler cannot check, prints the findings — and exits non-zero on any failure (no key, no diff, an HTTP error, an empty answer), because a review that could not run must go red, not green.

DIY 3: A review step that cannot lie

1. Add the free API key from the RAG lab (or make one at aistudio.google.com/apikey) to your copy of the repo as a repository secret named LLM_API_KEY (Settings → Secrets and variables → Actions).

2. Add a second workflow, .github/workflows/review.yml, that runs on pull requests, diffs the branch against its base, and runs the starter on that diff:

   name: review
   on: [pull_request]
   jobs:
     review:
       runs-on: ubuntu-latest
       permissions:
         contents: read
         pull-requests: write
       steps:
         - uses: actions/checkout@v7
           with:
             fetch-depth: 0
         - name: Review the diff, and fail if the review could not run
           env:
             LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
           shell: bash
           run: |
             git fetch --quiet origin ${{ github.base_ref }}
             git diff origin/${{ github.base_ref }}...HEAD > pr.diff
             python lectures-and-labs/week10/cicd_lab/review_diff.py pr.diff | tee review.md
         - name: Post the findings as a comment
           env:
             GH_TOKEN: ${{ github.token }}
           run: gh pr comment ${{ github.event.pull_request.number }} --body-file review.md
   

3. It posts its findings as a comment, not a commit — the last step above. Keep it that way.

4. Open a pull request with a deliberately badly-named function and confirm the review notices.

5. Now break the secret on purpose — rename it, or set it to nonsense — and confirm the job goes red rather than reporting a clean review.

What you should have

A review job that comments on pull requests, and evidence from step 5 that a broken run goes red instead of green.

4. Evals: checking the unrepeatable

summarise.py in this folder wraps a small summarising feature. Same input, different output. Everything you know about testing assumes that cannot happen.

DIY 4: Watch assertEqual fail on a good answer

1. In tests/test_summary.py, write test_exact: a test asserting the summary of samples/article_1.txt equals a fixed string you got from one run.
2. Run it. It almost always fails at once; if it passes, that was luck.
3. Run it again. And again.
4. Record what happens and why.

Expected output

FAILED tests/test_summary.py::test_exact
  AssertionError: assert 'Revenue rose 12% on renewals.'
                      == 'Revenue grew 12% year on year.'

DIY 5: Climb the assertion ladder

Replace that test with property checks that are true of any acceptable summary.

1. Build a set of at least eight cases in evals/cases.json, each with an input and the properties its output must satisfy.
2. Write evals/run_evals.py to score the set and print a pass rate, not pass/fail.
3. Use the strongest rung each property allows:
   • non-empty · under a length limit · mentions the key figure · says which way the figure moved · does not echo the prompt back
4. Run it and record the rate.

Expected output

Running 8 cases

  case-01  PASS
  case-02  PASS
  case-03  FAIL  (missing key figure: "12%")
  ...

Pass rate: 7/8 (87.5%)

DIY 6: Prove a change helped

1. Note your current pass rate.
2. Change PROMPT at the top of summarise.py — try to improve it. The stand-in model understands a short list of instructions, documented at the top of that file: mention the figure, mention the direction, set a word limit, ask for two sentences. A real model understands far more, far less predictably — which is why you measure.
3. Re-run the whole set, more than once.
4. Record the before and after rates.
5. Answer honestly: did it improve, get worse, or move cases around?

What you should have

Two pass rates and an honest verdict, including any case that got worse while others improved.

Common mistakes

• Adding a test suite before secret scanning. Order by what you cannot undo.
• Never watching the pipeline fail, so you have no evidence it works.
• Letting a failed model call report "no findings" — a green tick over a check that never ran is worse than no check.
• Asserting equality on non-deterministic output, then concluding the feature is broken.
• Reaching for a judge where a property check would have been exact, cheap and deterministic.
• Tuning a prompt against the cases you happened to look at.

Summary

• A pipeline is a script with a trigger; the value is that it runs when you forget.
• Order checks by irreversibility — secret scanning beats a test suite for value per minute.
• Ask AI what a compiler cannot check, and make a failed call a failure.
• Non-deterministic features need a set and a rate, not an assertion.
• Climb the ladder: exact → contains → structural → property → judge. Use the strongest rung the task allows.
• Without a measured set, prompt tuning is folklore.