Repository navigation
Add opt-in AI output threat model - #22810
Conversation
jacknojo
left a comment
jacknojo
left a comment
There was a problem hiding this comment.
Thanks Owen, made two followup commits for your comments.
There was a problem hiding this comment.
🟢 Approval recommended
The shared configuration, documentation, and focused regression coverage consistently implement the described opt-in behavior.
0 open findings
What changed in this PR
Adds the shared, opt-in ai threat model for LLM-generated output.
Changes:
- Registers
aias a known threat-model kind. - Documents the new kind and adds a change note.
- Tests explicit enablement while preserving default-disabled behavior.
| File | Description |
|---|---|
shared/threat-models/ext/threat-model-grouping.model.yml |
Registers ai under all. |
shared/threat-models/change-notes/2026-10-12-ai-threat-model.md |
Announces the opt-in threat model. |
javascript/ql/test/library-tests/threat-models/ai-enabled/ActiveAiThreatModel.ql |
Verifies ai is known and active. |
javascript/ql/test/library-tests/threat-models/ai-enabled/ActiveAiThreatModel.ext.yml |
Enables ai for the regression test. |
javascript/ql/test/library-tests/threat-models/ai-enabled/ActiveAiThreatModel.expected |
Records the expected test result. |
docs/codeql/reusables/threat-model-description.rst |
Documents the ai category. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Register ai as a shared threat model, document its scope, and add a regression test for explicit enablement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b1133f4-388a-4e13-a00f-dce2b41cd295
c2fa18d to
9ce464a
Compare
|
The integration tests failures are surely spurious. I think you can just ignore them and merge this PR. Did you rebase and force push to get CI to run again? You can just re-run those two tests, either manually or by writing |
I did, I figured it might have been something solved in |
Callback arguments and low-level requests are produced by a language model, so they are sources of the opt-in ai threat model added in github#22810. Request headers and bearer tokens come from the HTTP client and stay sources of the remote threat model.
Handler parameters are produced by a language model, so they are sources of the opt-in ai threat model added in github#22810. HTTP headers and bearer tokens come from the HTTP client and stay sources of the remote threat model.
github/codeql#22822 makes py/xxe, py/xml-bomb and py/nosql-injection see data-extension sources; the upstream proposals now use the opt-in ai threat model (github/codeql#22810) for handler input. The packs in this repository stay as measured.
Summary
Introduce the shared
aithreat-model kind for output generated by large language models (LLMs). This establishes the configuration support for follow-up source models and queries, without adding third-party library models or new queries.aiunderall, making it a known, independently configurable threat-model kind. It remains disabled by default.codeql/threat-models.aiand requires bothknownThreatModel("ai")andcurrentThreatModel("ai")to hold. The existing default-configuration test confirms thataiis not enabled by default.No new QL framework logic is necessary: the existing threat-model implementation derives known kinds and configuration behavior from the shared grouping table.
Validation
Both tests passed with the CodeQL CLI:
codeql test run \ javascript/ql/test/library-tests/threat-models/ai-enabled \ javascript/ql/test/library-tests/threat-models/default