Skip to content

Part 8 — AI & LLM features

AI-ID Feature Model (pinned) Inputs (data class) Output Human in the loop Fallback Est. cost / 1k calls Eval metric
AI-01 Daily standup {{model-id}} Task titles and status (C2, minimised) Markdown summary (JSON schema) User reviews before sharing “Generate failed. Retry” (S3) {{£}} Format validity ≥ 99%, factual errors ≤ 2%
AI-02 Task triage suggestions {{model-id}} Task title and description Tags, priority, assignee suggestion User accepts or rejects Manual triage {{£}} Acceptance rate ≥ {{60%}}
  • Lawful basis: usually contract (the feature the user asked for) or legitimate interests (9.4). Optional AI on sensitive content needs consent and a DPIA.
  • Provider terms:
    • A signed DPA.
    • No training on your data.
    • A stated prompt retention period (aim for zero or the minimum).
    • A processing region.
    • Sub-processors listed (3.6). Recorded in ADR-006.
  • Minimise and redact:
    • Send only the fields the task needs.
    • Replace names and emails with placeholders ([PERSON_1]) and rehydrate them server-side.
    • Never send C3 data without an approved DPIA.
  • Storage:
    • AI output is user content, kept and deleted with its parent record.
    • Prompt and response logging is off by default; if enabled for debugging, keep it ≤ 30 days.
    • ai_generations records usage only.
  • Transparency: the privacy notice names the AI provider, its purpose and any international transfer (9.6, 9.17).
  • Prompts live in lib/ai/prompts/*.ts with a PROMPT_VERSION, are reviewed like code and are recorded in each ai_generations row.
  • Structured output is validated with Zod. On failure, retry once with the validation error, then show S3.
  • Set maxOutputTokens, a timeout ({{30 s}}) with a user-facing Cancel button (S14), and up to 2 retries on 429/5xx with jittered backoff.
  • Stream long outputs, but render them as sanitised Markdown only.

8.4 Safety & security (OWASP Top 10 for LLM Applications)

Section titled “8.4 Safety & security (OWASP Top 10 for LLM Applications)”
  • Keep system instructions separate from untrusted content, and wrap user or retrieved text in clear delimiters. Instructions found inside content are data, not commands.
  • Model output never directly executes SQL, shell commands or HTML, and never triggers an action. Tool calls go through server-side authorisation and explicit user confirmation for anything with side effects.
  • Moderate user-facing generation where misuse is plausible. Include jailbreak and injection cases in the eval set.
  • Quotas and spend caps prevent cost exhaustion (8.7).
  • Agents that act, in the product or as the coding agent that builds it, get the least access that works, short-lived credentials, and a person’s approval for anything irreversible (12.1). The OWASP Top 10 for Agentic Applications lists the risks (Appendix B).
  • Keep a golden dataset of ≥ {{30}} representative cases, including edge cases: empty input, very long input, other languages, injection attempts.
  • Set acceptance thresholds per metric (8.1). Run the eval suite in CI whenever the prompt, model or parameters change. Block the merge on regressions.
  • Early on, review a weekly sample of {{20}} real outputs (pseudonymised) and feed what you find back into the dataset.
  • Label AI output in the UI: “AI-generated — check before sharing”. Make it editable before it’s used.
  • EU AI Act Art. 50 (applicable since 2 Aug 2026):
    • Tell people when they are interacting with an AI system, unless it’s obvious.
    • Generated synthetic audio, images, video or text must be marked in a machine-readable way. Systems already on the market before 2 Aug 2026 have until 2 Dec 2026 to comply with this marking duty.
    • Deployers of deepfakes and of AI-generated text published to inform the public have disclosure duties.
  • AI literacy (Art. 4, as amended by the Digital Omnibus on AI, Reg. (EU) 2026/1744, in force 27 Jul 2026): take measures that support AI literacy among the people who operate AI systems for you. No specific individual level has to be guaranteed.
  • High-risk check:
    • Using AI for recruitment, worker management or monitoring, education access or scoring, credit or essential services, or biometrics may make the system high-risk (AI Act Annex III). The Digital Omnibus on AI deferred these obligations to 2 Dec 2027.
    • Get advice before building in these areas.
  • Automated decisions with legal or similarly significant effects follow 9.22, including human review.
  • Nigeria: before deploying emerging technologies such as AI, GAID Art. 43 expects three things: a DPIA (filed with the NDPC as part of the compliance audit return), an assessment of whether to test first in a low-risk environment, and continuous monitoring once deployed. NDPA s.37 gives rights over automated decisions.
  • Per-user and per-workspace quotas; a monthly spend cap with alerts at 50/80/100%; hard stop at the cap.
  • Cache identical requests where that’s safe. Use the smallest model that passes the evals.
  • Wrap AI behind a feature flag so the rest of the app works without it (S15). Configure a fallback model only after it passes the same evals.