AI Agents for Software and Systems Security

Course Description

AI agents are becoming both tools for security analysis and components of the software systems that security must protect. This course studies both roles through one organizing question: what evidence justifies an agent’s security claim or action? Students build and instrument tool-using agents, connect them to static and dynamic analyses, evaluate them on held-out vulnerability cases, and examine how untrusted content, tools, credentials, and generated code can drive unsafe actions in AI-integrated systems.

Five sandboxed labs lead to an individual Evidence Audit and a team project in analysis, attack, or defense. Across all work, students must provide a reproducible trigger, an explicit threat model, and a clear statement of limitations. The course emphasizes framework-independent principles rather than any single model or agent platform, using current research, public benchmarks, CVEs, and patches as case material.

Curricular Role

This course complements advanced offerings in software security, systems, trustworthy AI, and privacy. It does not assume prior machine-learning or security coursework; instead, it teaches students how to connect model-based reasoning to program evidence, reproducible evaluation, and enforceable authority boundaries. It can serve as an advanced elective or as a bridge between cybersecurity and AI curricula.

Prerequisites & Bridge

Security courses often assume a background — C, assembly, debuggers, or competition experience — that students have had very different opportunities to build. This course states its prerequisites explicitly and provides bridge material for the rest. Prior background is a starting point, not a measure of ability. Students may arrive from computer science, information science, analytics, policy, or related programs; the course is designed to be completed on the stated prerequisites alone.

Required

• Data structures and one systems course, such as operating systems, computer organization, or systems programming.
• Reading fluency in C and working fluency in Python.
• Comfort at a Unix command line and with git.

Not assumed, and not required

• Prior security coursework, competition experience, or exploit development.
• Machine-learning coursework or experience with LLM APIs, prompting, or agent frameworks.
• Writing assembly. Students read disassembly and stack layouts; the bridge module covers what is needed.

Week 0 bridge (ungraded, complete by Week 3)

A short self-check routes each student to two self-paced modules: (a) memory, stack and heap layout, and inspecting live program state under GDB, and (b) calling a model, reading a tool-call trace, and reasoning about a context window. Both ask students to predict program behavior first and then check the prediction — the procedure used throughout the course. The modules remain available all semester, and office hours in the first three weeks prioritize bridge questions.

Differentiated Undergraduate and Graduate Expectations

All students complete the same core labs and demonstrate the same learning outcomes. Undergraduate students may build from course-provided program analyses, agent scaffolds, and evaluation harnesses. Graduate students additionally reproduce or extend one research result, implement a substantive baseline or ablation in Lab 2 or the final project, and submit an expanded analysis of related work and limitations. Graduate-only criteria are identified explicitly in the relevant rubrics rather than added as unrelated assignments.

Learning Outcomes

By the end of the course, students will be able to:

1. CLO 1. Analyze the architecture of a tool-using agent and explain how design choices create characteristic failure modes.
2. CLO 2. Decompose a security task into bounded steps and justify which steps require deterministic analysis, model judgment, or execution.
3. CLO 3. Validate a security claim with a reproducible trigger, an explicit threat model, documented assumptions, and a clear statement of limitations.
4. CLO 4. Evaluate an agent or defense on held-out tasks using effectiveness, cost, and run-to-run variance.
5. CLO 5. Trace untrusted influence and delegated authority from inputs through tools, code changes, and runtime effects, then implement a control that blocks a concrete consequence while preserving legitimate work.
6. CLO 6. Conduct authorized, isolated, and reproducible security experiments and communicate positive, negative, and inconclusive results responsibly.

Modules

Module 1 (Weeks 1–4) — Agents, Analyses, and What Counts as Evidence

Students study the anatomy of a tool-using agent and the program analyses it can call. They build a minimal agent, deliberately alter one component at a time, and connect observed failures to design choices. In parallel, the course establishes its evidence standard: what distinguishes a security claim that another person can check from one that rests on a model’s assertion.

Module 2 (Weeks 5–8) — AI for Security: Discovery, Patches, and Validation

Students use agents as bounded analysis modules. They derive detection rules from vulnerability-fixing history, narrow rules to candidate sites with static analysis, evaluate each site, and validate surviving findings through execution. The module then asks whether a patch removes the root cause and whether a benchmark measures analysis rather than memorization.

Module 3 (Weeks 9–12) — Security of AI-Integrated Systems: Data to Action

Students treat agents as attack surface. A comment, retrieved document, tool description, or third-party configuration can become an instruction that changes code or runtime behavior. Students attack a provided agent pipeline through its supply chain, trace the path from information to action, and evaluate least-authority controls at the boundary.

Module 4 (Weeks 13–15) — Disclosure, Design Review, and Project

Students examine coordinated disclosure, authorization, applicable human-participant review and consent procedures, and the burden a report places on maintainers. Teams then defend their threat model and evidence plan in rotating design reviews and present a reproducible result, including what they could not establish.

Weekly Schedule

Calendar dates, holidays, and the guest session are set per offering. Most weeks assign one required reading; a second reading is recommended or required for graduate students.

Assessment

Every graded component maps to at least one learning outcome, and every outcome is assessed more than once. There is no exam; the individual Evidence Audit serves the role of a midterm.

The five labs

• Lab 0 — Sandbox and trace (2%, completion-based). Start the course container, run a provided agent against a toy target, and produce a complete, replayable tool-call trace.
• Lab 1 — Instrument the loop. Build a minimal tool-using agent with three tools. Then remove memory, shrink the context budget, and withhold one tool, and report which failure each change produces. (CLO 1)
• Lab 2 — Bounded adjudication. Given a detection rule, a course-provided candidate set, and a baseline pipeline, modify the agent’s adjudication strategy. Report precision and recall against held-out labels, cost per site, and variance across repeated runs. Graduate students additionally implement or compare a substantive alternative baseline or ablation. (CLO 2, 4)
• Lab 3 — Injection through the supply chain. In a provided agent-integrated repository, place untrusted content in a code comment, README, test fixture, or tool description and demonstrate an action the agent should not take. Trace the path from content to action. (CLO 5)
• Lab 4 — Least authority. Add a control to the Lab 3 pipeline. Demonstrate the consequence it blocks, measure false blocking on legitimate tasks, and state what it does not defend against. (CLO 5)

Evidence Audit

Each student receives a program snapshot and a candidate finding produced by an agent and must answer two questions: is the behavior real, and does it matter? The deliverable is a reproducible trigger — or a supported demonstration that the behavior cannot be triggered — together with a threat-model argument, an explicit list of assumptions, and an assessment of a proposed patch.

To reduce the chance that memorized public fixes can substitute for analysis, the audit uses a frozen course-provided model with a documented data cutoff. Cases either postdate that cutoff or are instructor-generated variants of known defects whose source mapping is withheld until grading. Roughly one-third of the distributed cases are not vulnerabilities; a well-supported negative verdict can earn full credit.

Final project

Teams of two or three choose one of three tracks and carry it to a defended result on course-provided targets. Teams are formed with instructor input to balance backgrounds, and the tracks reward different mixes of building, measurement, and analysis.

• Analysis track. Build an agent-native analysis for a specific bug class and evaluate it on a held-out set.
• Attack track. Take a provided agent-integrated system and establish a data-to-action path end to end, with a working demonstration in isolation.
• Defense track. Implement a control at an information-to-action boundary and evaluate it against both attacks and legitimate workflows.

Deliverables are a reproducible artifact (code, container, and one-command replay), a 6–8 page report, and an eight-minute presentation followed by a four-minute demonstration or question period. Each student also submits a one-page contribution and limitations memo or completes a brief oral defense. Presentations use Week 15 and the scheduled final period; larger sections use a poster-and-demo format. Projects are graded on design, evidence, reproducibility, and limitations — not on whether a vulnerability was found.

Evidence rubric

Applied to every graded security claim from Lab 2 onward.

• Trigger. A concrete input, state, or execution that produces the claimed behavior — or a supported argument that none exists. Model output is not a trigger.
• Threat model. Who the attacker is, what they control, and which trust boundary the behavior crosses. A behavior outside the project’s threat model is a finding about the threat model, not necessarily about the code.
• Reproducibility. Another person following the submission reaches the same result. Versions are pinned, the environment is captured, and non-determinism is reported.
• Limits. What the evidence does not establish: paths not covered, configurations not tested, and assumptions that would invalidate the claim.

Negative results

A rigorous negative result can earn full credit when the submission characterizes the search space, explains why the method should have found the target if present, and states what the absence of a result does and does not support. An unsupported positive claim cannot earn equivalent credit.

Targets & Scope Policy

All graded work — the labs, Evidence Audit, and final project — runs on course-provided targets: purpose-built programs, containerized agent pipelines, and self-hosted snapshots of open-source projects whose relevant defects are already public and fixed upstream. This preserves the technical substance of vulnerability analysis without consuming maintainer attention. Students may perform only the analysis and controlled exploitation specified by the assignment, within the sandbox and under the minimum-necessary-access rule below.

Self-hosting rules

• Target services and project snapshots run only inside the course sandbox.
• Network egress is blocked except for institutionally approved package mirrors and, when used, approved model endpoints.
• No interaction with an upstream project’s infrastructure, issue tracker, users, or maintainers.

If you find something new

Because snapshots come from real projects, analysis may occasionally surface a defect that does not correspond to a known vulnerability. Stop the analysis, preserve the evidence, and bring it to the instructor. Any coordinated disclosure is handled through faculty and institutional processes; students do not file reports, open issues, or contact maintainers as part of coursework. Establishing that a candidate is not a vulnerability is also a valid outcome under the evidence rubric.

Beyond provided targets

Live and third-party systems are out of scope. Work on any target outside the provided set requires prior written approval, isolated execution, and instructor review, and it is never necessary to earn full marks.

Policy on AI Use

Agents are the subject of this course, so using them is expected rather than restricted. The course requires provenance, verification, and individual ownership of submitted claims.

• Declare it. Every submission includes an AI-USE.md recording which models and tools were used, what they were asked to do, and which claims the student verified and how.
• Own it. Students are responsible for every submitted claim, whatever produced it. An unverified model assertion presented as a finding is graded as an unsupported claim, not automatically as misconduct.
• Misconduct is different. Presenting another person’s analysis or a published write-up as one’s own, fabricating results or traces, or falsifying an AI-USE.md are academic-integrity violations.
• Traces are part of the work. For labs, the tool-call trace is a graded artifact. It must show what the agent actually did, including failed runs.

Compute & Cost

The course is designed to run on institutionally approved containers or virtual machines. Each student’s environment is isolated and reproducible, so in-person, hybrid, and online offerings use the same targets, artifacts, and evidence standards.

All core assessments can be completed with a course-provided open-weight model; no student must purchase API access or use a particular commercial service. When resources permit, metered commercial APIs are provided only for controlled comparison exercises. In Lab 2, all students report cost per site and run-to-run variance for their pipeline, and graduate students additionally compare a substantive alternative baseline or model configuration on the same candidate set. No assessment depends on continued access to a specific vendor or model version.

Representative Readings

No textbook. Most weeks assign one required reading, with a second reading recommended or required for graduate students. Readings mix research papers, technical reports, and primary documentation; the list is refreshed for each offering.

Module 1 — Agents and analyses

• Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. [paper] [arXiv]
• Schick et al., Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023. [paper] [arXiv]
• Lu, Pakki, and Wu, Detecting Missing-Check Bugs via Semantic- and Context-Aware Criticalness and Constraints Inferences. USENIX Security 2019. [paper]
• Model Context Protocol, Specification, revision 2026-07-28. Primary documentation. [spec]

Module 2 — AI for security

• Wu et al., Precisely Characterizing Security Impact in a Flood of Patches via Symbolic Rule Comparison. NDSS 2020. [paper]
• Wu, Xiao, Liao, and Lu, OS-Aware Vulnerability Prioritization via Differential Severity Analysis. USENIX Security 2022. [paper]
• Jimenez et al., SWE-bench: Can Language Models Resolve Real-world GitHub Issues? ICLR 2024. [paper] [arXiv]
• Zhang et al., Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR 2025. [paper] [arXiv]
• Glazunov and Brand, Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models, and Big Sleep team, From Naptime to Big Sleep: Using Large Language Models to Catch Vulnerabilities in Real-World Code. Google Project Zero blog, 2024. [Naptime] [Big Sleep]
• DARPA AI Cyber Challenge (AIxCC) Final Competition, DEF CON 33 — finalist cyber reasoning systems (open-source releases) and the ATLANTIS technical report. 2025. [AIxCC] [systems] [ATLANTIS]

Module 3 — Security of AI-integrated systems

• Greshake et al., Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec 2023, co-located with ACM CCS. [paper] [arXiv]
• Debenedetti et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024, Datasets and Benchmarks Track. [paper] [arXiv]
• Debenedetti et al., Defeating Prompt Injections by Design (CaMeL). arXiv preprint, 2025. [arXiv] [code]
• Bai et al., APILOT: Improving the Security and Usability of LLM Code Suggestions via Outdated API Mitigation. ACSAC 2025. [paper] [code]
• OWASP, Top 10 for LLM Applications. 2026 edition. [document]
• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, July 2024. [document]

Module 4 — Disclosure

• Linux kernel documentation, Researcher Guidelines. Kernel documentation. [document]
• Householder et al., The CERT Guide to Coordinated Vulnerability Disclosure. CERT/CC web edition; selected chapters. [document]
• Case studies in disclosure failures, corrective practices, and maintainer burden. Week 13.

Course Policies

Institution-specific language on accessibility, academic-integrity procedures, counseling and health resources, religious observance, Title IX, and related requirements would be inserted from the host department. The policies below are specific to this course design.

Late work

Each student has a budget of 96 grace hours for the semester, spent in whole hours across labs and reading responses, with no penalty and no explanation required. After the budget is exhausted, late work loses 20% per 24 hours, with a hard stop 72 hours past the deadline. Grace hours do not apply to the Evidence Audit, design review, or presentation, which are scheduled events. Extensions for illness, family circumstances, or other disruption are handled separately and do not consume the budget.

Attendance and participation

Attendance is not scored. Participation is assessed through specific acts: contributing substantive feedback in the two Week 14 design-review sessions and participating in the Week 13 disclosure discussion. Both have make-up paths for students who cannot attend a given session. Missing lab sessions is costly in practice because that is where tooling and environment problems are resolved.

Collaboration

Labs and the Evidence Audit are individual: students may discuss approaches but submit their own code, traces, and analysis. The project is collaborative; the final report identifies each member’s contribution, and each student submits an individual contribution and limitations memo or completes a brief oral defense.

Grading scale

A ≥ 93 · A− ≥ 90 · B+ ≥ 87 · B ≥ 83 · B− ≥ 80 · C+ ≥ 77 · C ≥ 70 · D ≥ 60 · F < 60

Office hours and access

Two scheduled hours per week plus appointments, offered both in person and online. Students are encouraged to bring a prediction and an observation that disagree; the mismatch is usually the most useful starting point for debugging.