Practical AI Security Engineering Program

PART 1: FOUNDATIONS & BUILDING AI SYSTEMS
• Module 1: Introduction to AI Security Engineering
  • AI Introduction: Types of AI learning algorithms, components of AI systems and Use Cases (Predictive, Generative and Agentic)
  • Model Development Lifecycle
  • NLP and Generative AI: Understanding Tokens, Context Windows, System Prompts, and Model Responses
  • Limitations of AI: hallucination, bias, explainability gaps, non-determinism, data privacy concerns, overreliance
  • Understanding How AI Applications Are Different from Traditional Applications
  • Overview of the AI Threat Landscape: emerging attack classes unique to AI systems
  • Introduction to AI Security Engineering: what the discipline covers and how it differs from traditional AppSec/InfoSec
  • AI Application Attack Surface Overview: prompt risk, data risk, model risk, API risk, pipeline risk, infrastructure risk
  • Defense-in-Depth for AI: introducing the layered model (data, model, application/pipeline, infrastructure, governance/monitoring)
  • Responsible AI Concepts and the Difference Between AI Safety, AI Security and AI Governance
  • Practical Exercises:
    • Mapping a simple AI application architecture
    • Identifying AI assets, entry points, and data flows in a sample system
    • Identifying where prompt injection and data leakage may occur in that system
• Module 2: ML System Architecture and Engineering
  • ML system architecture: data pipeline, feature store, training pipeline, model registry, serving/inference layer, monitoring layer
  • ML lifecycle stages: collection→ preprocessing→ training→ validation→ deployment→ monitoring→ retraining
  • Trust boundaries applied to ML systems: data, model, pipeline, infrastructure boundary
  • Python and core ML libraries overview: NumPy, Pandas, Scikit-learn, TensorFlow/Keras, NLTK
  • Common security data formats: network logs, Windows Event Logs, email headers, binary file features
  • Basic feature extraction concepts: flow statistics, text n-grams, byte histograms
  • Practical Exercises:
    • Diagram the architecture and trust boundaries of a sample ML-based security pipeline (e.g., a network intrusion detector)
    • Build a minimal scikit-learn pipeline (load→ preprocess→ train→ evaluate→ save artifact).
• Module 3: Building ML and DL Security Controls
  • MITRE ATT&CK and the Pyramid of Pain as a guide for detector design
  • Network attack detection: flow-based feature engineering and classification
  • Phishing email detection: TF-IDF vectorisation and text classification
  • Malware detection: static file features and deep learning classification
  • User behaviour anomaly detection: UEBA concept, unsupervised anomaly scoring
  • Serving trained models as REST APIs with FastAPI
  • Experiment tracking with MLflow
  • Practical Exercises:
    • Network Attack Detection: feature engineering→ training→ evaluation→ FastAPI serving→ MLflow registration
    • Phishing Email Detection: TF-IDF classifier, evaluated and served via FastAPI
    • Malware Detection: static-feature DL classifier for malware
• Module 4: LLM and Agentic AI Architecture and Engineering
  • LLM application components: API layer, model provider, vector database, retrieval layer, logging layer
  • Local and API-based LLM setup: OpenWebUI, Ollama, LM Studio, OpenRouter — cost and data-retention tradeoffs
  • Prompt engineering essentials: zero-shot, few-shot, chain-of-thought, system prompt design
  • RAG architecture: ingestion, chunking, embeddings, vector storage, retrieval, generation
  • Vector stores: ChromaDB, FAISS and retrieval quality as a security concern
  • Parameter-efficient fine-tuning
  • Fine-tuning vs RAG vs prompt engineering: cost, data, and latency tradeoffs
  • Agent architecture: planning, memory, tool use, external API access, action execution, feedback
  • Agentic frameworks and concepts: LangChain/LangGraph, CrewAI/A2A overview, MCP and tool exposure, context-overflow concept, markdown-native agent architecture
  • Trust boundaries for LLM/Agentic systems: model, tool, memory/context, retrieval boundary
  • Practical Exercises:
    • Set up Open WebUI with local Ollama/OpenRouter and a security-analyst system prompt
    • Fine-Tuning Walkthrough: fine-tune a small open-source model on a security dataset, compare before/after responses
    • RAG Pipeline: build a RAG pipeline over a threat-intel knowledge base
    • MCP Exploration: connect to a pre-built MCP server and observe tool exposure

PART 2: ATTACKING AI SYSTEMS

• Module 5: AI Threat Modelling
  • Threat modeling fundamentals: threat actors, assets, abuse cases, entry points, data flows, security controls
  • Applying MITRE ATLAS and STRIDE to AI applications
  • Revisiting trust boundaries in AI Systems with adversarial intent in mind
  • Practical Exercises:
    • Threat-model the ML pipeline built in Module 3 and the RAG/agentic application built in Module 4- produce trust-boundary/data-flow diagrams
• Module 6: Adversarial ML: Attacking ML and DL Systems
  • OWASP ML Security Top 10
  • Adversarial drift
  • Poisoning, evasion, extraction, model inversion, membership inference
  • Attack tooling: Adversarial Robustness Toolbox (ART), used as a black-box tool
  • Practical Exercises:
    • Data Poisoning Simulation: inject crafted samples into the training pipeline, retrain, observe targeted accuracy impact
    • Evasion Attack: run a pre-built ART evasion attack against the intrusion classifier, interpret accuracy degradation, document using ATLAS
• Module 7: Adversarial Attacks on LLMs and Agents
  • OWASP LLM Top 10, OWASP Agentic AI Top 10, MAESTRO (threat model for agentic systems)
  • Direct and indirect prompt injection; system prompt leakage; jailbreaking techniques
  • Context-window overflow and vector store poisoning
  • Tool misuse and excessive agency, multi-agent trust exploitation
  • MCP-specific attack vectors: tool output injection, privilege escalation
  • Automated testing tools: Garak, PyRIT, Promptfoo
  • Practical Exercises:
    • Automated vulnerability scan(Garak/Promptfoo) against an AI application and review findings against OWASP LLM Top 10
    • Attack a pre-built, deliberately vulnerable agentic application: direct/indirect injection, memory poisoning; map findings to OWASP Agentic Top 10 and MAESTRO layers

PART 3: DEFENDING AI SYSTEMS

• Module 8: AI Data Security
  • The AI data lifecycle: collection, labeling, preprocessing, training, validation, testing, inference, output data
  • Prompt data, retrieval data, vector database data, logs, and telemetry
  • Data classification for AI: public, internal, confidential, restricted, personal, sensitive, credential
  • Data leakage patterns: prompt leakage, file upload leakage, model output leakage, log leakage, vector database leakage
  • RAG over-retrieval risk and copilot-style oversharing risk
  • Source code and secret exposure risk in AI workflows
  • Data sanitization and de-identification: anonymization, pseudonymization, minimization, masking, redaction
  • Data lineage and provenance
  • Privacy preserving learning techniques
  • Practical Exercises:
    • PII detection and masking using Microsoft Presidio
    • Prompt sanitization techniques— build a safe AI input pipeline
    • Secrets detection using Gitleaks/TruffleHog
    • Classify sample AI data and review AI logs for sensitive data exposure
• Module 9: AI Model Security (ML and DL)
  • Adversarial training: hardening models with adversarial examples
  • MLflow Model Registry: model versioning and model cards
  • Model artifact protection: safe serialisation vs. unsafe formats
  • Model integrity validation: hashing, signing, checksums
  • Registry protection: access control, approval tags, rollback planning
  • Practical Exercises:
    • Retrain the Module 3 classifier with adversarial examples and measure robustness improvement
    • Register the model in MLflow with a version tag, and auto-generate a model card
    • Validate the model artifact’s hash and serialisation format
• Module 10: AI Model Security (LLMs and Agentic AI)
  • Input guardrails: prompt injection detection, PII redaction, topic filtering, length limits
  • Output guardrails: hallucination detection, sensitive content filtering, format enforcement
  • Guardrail approaches: rule-based, classifier-based, LLM-as-judge
  • Open-source guardrail tools: LLM-Guard, LlamaGuard, GuardrailsAI, NeMo Guardrails
  • LLM gateway: rate limiting, authentication, request logging, cost controls
  • Model and artifact integrity for LLMs
  • Harness engineering overview: tool sandboxing, behavioural constraints, human oversight gates
  • Practical Exercises:
    • Wrap the insecure LLM application with input/output guardrail scanners and harden system prompts
• Module 11: AI Application & API Security
  • AI AppSec vs traditional AppSec
  • AI APIs and LLM API workflows
  • API authentication and authorization for AI applications, token handling and key protection
  • Runtime protections: Web Application Firewalls (WAFs) and inference-stage rate-limiters to mitigate DoS and malicious query floods
  • RBAC/ABAC, object-level authorization
  • Document-level authorization in RAG systems
  • Tool-level authorization in AI agents
  • Secure design for AI plugins, connectors, and MCP integrations
  • Practical Exercises:
    • Review an AI API for missing authentication/authorization (Postman/Burp Suite)
    • Identify broken access control in a RAG scenario
    • Test for API key exposure and weak token handling

PART 4: AI SECURITY OPERATIONS

• Module 12: Secure MLOps: CI/CD Pipeline Security
  • MLOps pipeline stages: data validation, training, evaluation, registration, deployment
  • Secure data pipeline design: Input validation and schema enforcement before training begins, dataset versioning and rollback using DVC
  • Pipeline access control and credential/secrets risk in build environments
  • CI/CD for AI: triggers on code, data, and model changes
  • Automated pass/fail security gates: SAST(Semgrep, CodeQL), secrets scanning, dependency scanning(Snyk, pip-audit)
  • Production promotion gates: performance and fairness thresholds before a model goes live
  • Practical Exercises:
    • Build a GitHub Actions pipeline combining data validation, training, evaluation, and SAST/secrets/dependency gates
    • Configure an MLflow promotion gate so a model only registers as “production” if it clears the performance/fairness threshold
    • Demonstrate a pipeline failure from a deliberately introduced issue and a clean run that promotes the model
• Module 13: AI Infrastructure, Access Control & Supply Chain Security
  • AI infrastructure components: compute/GPU workloads, containers, model-serving endpoints, vector database storage
  • Identity and Access Management (IAM) for AI infrastructure: RBAC and least privilege for dataset modification, model deployment, and registry/tuning-parameter access
  • Environment hardening: network segmentation, GPU cluster isolation, and lateral-movement prevention
  • Container security for AI workloads
  • Open-source model and public dataset supply chain risk, third-party AI API risk
  • Model provenance verification
  • SBOM and AI-BOM concept: what they capture and why they matter for AI supply chain transparency
  • Practical Exercises:
    • Deploy an LLM gateway(LiteLLM) in front of the application, configured with rate limits, authentication, and request logging
    • Scan a container image for vulnerabilities(Trivy)
    • Using Model Provenance Toolkit
    • Generate an SBOM/AI-BOM for a sample AI app using a guided template (Syft + CycloneDX)
• Module 14: AI Security Monitoring and Incident Response
  • ML drift monitoring: data drift vs. concept drift, reviewing an Evidently AI drift report
  • AI security monitoring overview: prompt, API usage, model query, data access, tool execution, and agent action monitoring
  • Output validation: monitoring deployed outputs for sensitive data leakage, copyright infringement, or malicious content generation
  • LLM observability tooling: Arize Phoenix and Langfuse/Langsmith for tracing calls, retrieval steps, tool invocations
  • Abuse detection: spotting attacker behaviour over time (high-volume querying, repeated extraction attempts, repeated jailbreak attempts)
  • Vulnerability intake, classification, and severity rating for AI issues
  • AI incident scenarios: prompt injection, sensitive data exposure, model extraction, poisoned dataset, unauthorized model replacement, API abuse, shadow AI
  • Incident response playbooks: quarantine, retrain, and rollback procedures for compromised models
  • Honeypots in AI Applications
  • Practical Exercises:
    • Review a pre-built Evidently AI drift report and configure a basic alert threshold
    • Analyze AI API abuse logs for extraction/jailbreak patterns
    • Building and deploying a honeypot in RAG Pipeline
• Module 15: AI Governance and Responsible AI Practice
  • NIST AI RMF mapping exercise (Govern, Map, Measure, Manage)
  • Responsible AI properties of controls built in the course
  • Secure AI SDLC: Acquire vs Build governance questions
  • AI-BOM’s role in governance and audit evidence
  • Practical Exercises:
    • Given a sample AI system(a built combination of one ML model and one LLM/RAG application from earlier in the course), learners:
      • Review an AI API for missing authentication/authorization (Postman/Burp Suite)
      • Identify broken access control in a RAG scenario
      • Test for API key exposure and weak token handling

• Foundational AI and cybersecurity knowledge. Preferably should have done Infosectrain’s AI Cybersecurity Foundation Program or should have equivalent knowledge.
• Comfortable working with the command line, configuration files, and gen-AI assisted coding.

• Security professionals moving into AI:
  • Security Engineers
  • Security Architects
  • Penetration Testers/Red Teamers
  • SOC Analysts
  • AppSec Engineers
  • DevSecOps Engineers
• AI and engineering professionals moving into security:
  • Data Scientists
  • ML Engineers
  • LLM Engineers
  • MLOps/Platform Engineers

After completion of this course, you will be able to:

• Build a security-first mental model of AI systems: attack surface, trust boundaries, and defense-in-depth.
• Build, attack, and secure both traditional ML/DL security controls and GenAI/LLM/agentic applications.
• Threat-model AI systems using STRIDE, MITRE ATLAS, OWASP ML/LLM/Agentic Top 10, and MAESTRO.
• Apply adversarial techniques like poisoning, evasion, extraction, prompt injection, jailbreaking, memory/tool poisoning against systems built in the course.
• Implement defense-in-depth across data, model, application, and infrastructure layers: guardrails, gateways, adversarial training, artifact integrity, access control, and supply chain checks.
• Secure the full MLOps/LLMOps pipeline: CI/CD security gates, dataset/prompt versioning, promotion gates, and rollback procedures.
• Monitor AI systems in production and respond to AI-specific incidents like drift, abuse, extraction, prompt injection, shadow AI.
• Map technical controls to NIST AI RMF and Responsible AI properties, and produce governance evidence a real audit or compliance function would ask for.