Practical AI Security Engineering Program

PART 1: FOUNDATIONS & BUILDING AI SYSTEMS

• • Module 1: Introduction to AI Security Engineering
    • AI Introduction: Types of AI learning algorithms, components of AI systems and Use Cases (Predictive, Generative and Agentic)
    • Model Development Lifecycle
    • NLP and Generative AI: Understanding Tokens, Context Windows, System Prompts, and Model Responses
    • Limitations of AI: hallucination, bias, explainability gaps, non-determinism, data privacy concerns, overreliance
    • Understanding How AI Applications Are Different from Traditional Applications
    • Overview of the AI Threat Landscape: emerging attack classes unique to AI systems
    • Introduction to AI Security Engineering: what the discipline covers and how it differs from traditional AppSec/InfoSec
    • AI Application Attack Surface Overview: prompt risk, data risk, model risk, API risk, pipeline risk, infrastructure risk
    • Defense-in-Depth for AI: introducing the layered model (data, model, application/pipeline, infrastructure, governance/monitoring)
    • Responsible AI Concepts and the Difference Between AI Safety, AI Security and AI Governance
    • Practical Exercises:
      • Mapping a simple AI application architecture
      • Identifying AI assets, entry points, and data flows in a sample system
      • Identifying where prompt injection and data leakage may occur in that system
  • Module 2: ML System Architecture and Engineering
    • ML system architecture: data pipeline, feature store, training pipeline, model registry, serving/inference layer, monitoring layer
    • ML lifecycle stages: collection→ preprocessing→ training→ validation→ deployment→ monitoring→ retraining
    • Trust boundaries applied to ML systems: data, model, pipeline, infrastructure boundary
    • Python and core ML libraries overview: NumPy, Pandas, Scikit-learn, TensorFlow/Keras, NLTK
    • Common security data formats: network logs, Windows Event Logs, email headers, binary file features
    • Basic feature extraction concepts: flow statistics, text n-grams, byte histograms
    • Practical Exercises:
      • Diagram the architecture and trust boundaries of a sample ML-based security pipeline (e.g., a network intrusion detector)
      • Build a minimal scikit-learn pipeline (load→ preprocess→ train→ evaluate→ save artifact).
  • Module 3: Building ML and DL Security Controls
    • MITRE ATT&CK and the Pyramid of Pain as a guide for detector design
    • Network attack detection: flow-based feature engineering and classification
    • Phishing email detection: TF-IDF vectorisation and text classification
    • Malware detection: static file features and deep learning classification
    • User behaviour anomaly detection: UEBA concept, unsupervised anomaly scoring
    • Serving trained models as REST APIs with FastAPI
    • Experiment tracking with MLflow
    • Practical Exercises:
      • Network Attack Detection: feature engineering→ training→ evaluation→ FastAPI serving→ MLflow registration
      • Phishing Email Detection: TF-IDF classifier, evaluated and served via FastAPI
      • Malware Detection: static-feature DL classifier for malware
  • Module 4: LLM and Agentic AI Architecture and Engineering
    • LLM application components: API layer, model provider, vector database, retrieval layer, logging layer
    • Local and API-based LLM setup: OpenWebUI, Ollama, LM Studio, OpenRouter — cost and data-retention tradeoffs
    • Prompt engineering essentials: zero-shot, few-shot, chain-of-thought, system prompt design
    • RAG architecture: ingestion, chunking, embeddings, vector storage, retrieval, generation
    • Vector stores: ChromaDB, FAISS and retrieval quality as a security concern
    • Parameter-efficient fine-tuning
    • Fine-tuning vs RAG vs prompt engineering: cost, data, and latency tradeoffs
    • Agent architecture: planning, memory, tool use, external API access, action execution, feedback
    • Agentic frameworks and concepts: LangChain/LangGraph, CrewAI/A2A overview, MCP and tool exposure, context-overflow concept, markdown-native agent architecture
    • Trust boundaries for LLM/Agentic systems: model, tool, memory/context, retrieval boundary
    • Practical Exercises:
      • Set up Open WebUI with local Ollama/OpenRouter and a security-analyst system prompt
      • Fine-Tuning Walkthrough: fine-tune a small open-source model on a security dataset, compare before/after responses
      • RAG Pipeline: build a RAG pipeline over a threat-intel knowledge base
      • MCP Exploration: connect to a pre-built MCP server and observe tool exposure

PART 2: ATTACKING AI SYSTEMS

• • Module 5: AI Threat Modelling
    • Threat modeling fundamentals: threat actors, assets, abuse cases, entry points, data flows, security controls
    • Applying MITRE ATLAS and STRIDE to AI applications
    • Revisiting trust boundaries in AI Systems with adversarial intent in mind
    • Practical Exercises:
      • Threat-model the ML pipeline built in Module 3 and the RAG/agentic application built in Module 4- produce trust-boundary/data-flow diagrams
  • Module 6: Adversarial ML: Attacking ML and DL Systems
    • OWASP ML Security Top 10
    • Adversarial drift
    • Poisoning, evasion, extraction, model inversion, membership inference
    • Attack tooling: Adversarial Robustness Toolbox (ART), used as a black-box tool
    • Practical Exercises:
      • Data Poisoning Simulation: inject crafted samples into the training pipeline, retrain, observe targeted accuracy impact
      • Evasion Attack: run a pre-built ART evasion attack against the intrusion classifier, interpret accuracy degradation, document using ATLAS
  • Module 7: Adversarial Attacks on LLMs and Agents
    • OWASP LLM Top 10, OWASP Agentic AI Top 10, MAESTRO (threat model for agentic systems)
    • Direct and indirect prompt injection; system prompt leakage; jailbreaking techniques
    • Context-window overflow and vector store poisoning
    • Tool misuse and excessive agency, multi-agent trust exploitation
    • MCP-specific attack vectors: tool output injection, privilege escalation
    • Automated testing tools: Garak, PyRIT, Promptfoo
    • Practical Exercises:
      • Automated vulnerability scan(Garak/Promptfoo) against an AI application and review findings against OWASP LLM Top 10
      • Attack a pre-built, deliberately vulnerable agentic application: direct/indirect injection, memory poisoning; map findings to OWASP Agentic Top 10 and MAESTRO layers

PART 3: DEFENDING AI SYSTEMS

• • Module 8: AI Data Security
    • The AI data lifecycle: collection, labeling, preprocessing, training, validation, testing, inference, output data
    • Prompt data, retrieval data, vector database data, logs, and telemetry
    • Data classification for AI: public, internal, confidential, restricted, personal, sensitive, credential
    • Data leakage patterns: prompt leakage, file upload leakage, model output leakage, log leakage, vector database leakage
    • RAG over-retrieval risk and copilot-style oversharing risk
    • Source code and secret exposure risk in AI workflows
    • Data sanitization and de-identification: anonymization, pseudonymization, minimization, masking, redaction
    • Data lineage and provenance
    • Privacy preserving learning techniques
    • Practical Exercises:
      • PII detection and masking using Microsoft Presidio
      • Prompt sanitization techniques— build a safe AI input pipeline
      • Secrets detection using Gitleaks/TruffleHog
      • Classify sample AI data and review AI logs for sensitive data exposure
  • Module 9: AI Model Security (ML and DL)
    • Adversarial training: hardening models with adversarial examples
    • MLflow Model Registry: model versioning and model cards
    • Model artifact protection: safe serialisation vs. unsafe formats
    • Model integrity validation: hashing, signing, checksums
    • Registry protection: access control, approval tags, rollback planning
    • Practical Exercises:
      • Retrain the Module 3 classifier with adversarial examples and measure robustness improvement
      • Register the model in MLflow with a version tag, and auto-generate a model card
      • Validate the model artifact’s hash and serialisation format
  • Module 10: AI Model Security (LLMs and Agentic AI)
    • Input guardrails: prompt injection detection, PII redaction, topic filtering, length limits
    • Output guardrails: hallucination detection, sensitive content filtering, format enforcement
    • Guardrail approaches: rule-based, classifier-based, LLM-as-judge
    • Open-source guardrail tools: LLM-Guard, LlamaGuard, GuardrailsAI, NeMo Guardrails
    • LLM gateway: rate limiting, authentication, request logging, cost controls
    • Model and artifact integrity for LLMs
    • Harness engineering overview: tool sandboxing, behavioural constraints, human oversight gates
    • Practical Exercises:
      • Wrap the insecure LLM application with input/output guardrail scanners and harden system prompts
  • Module 11: AI Application & API Security
    • AI AppSec vs traditional AppSec
    • AI APIs and LLM API workflows
    • API authentication and authorization for AI applications, token handling and key protection
    • Runtime protections: Web Application Firewalls (WAFs) and inference-stage rate-limiters to mitigate DoS and malicious query floods
    • RBAC/ABAC, object-level authorization
    • Document-level authorization in RAG systems
    • Tool-level authorization in AI agents
    • Secure design for AI plugins, connectors, and MCP integrations
    • Practical Exercises:
      • Review an AI API for missing authentication/authorization (Postman/Burp Suite)
      • Identify broken access control in a RAG scenario
      • Test for API key exposure and weak token handling

PART 4: AI SECURITY OPERATIONS

• Module 12: Secure MLOps: CI/CD Pipeline Security
  • MLOps pipeline stages: data validation, training, evaluation, registration, deployment
  • Secure data pipeline design: Input validation and schema enforcement before training begins, dataset versioning and rollback using DVC
  • Pipeline access control and credential/secrets risk in build environments
  • CI/CD for AI: triggers on code, data, and model changes
  • Automated pass/fail security gates: SAST(Semgrep, CodeQL), secrets scanning, dependency scanning(Snyk, pip-audit)
  • Production promotion gates: performance and fairness thresholds before a model goes live
  • Practical Exercises:
    • Build a GitHub Actions pipeline combining data validation, training, evaluation, and SAST/secrets/dependency gates
    • Configure an MLflow promotion gate so a model only registers as “production” if it clears the performance/fairness threshold
    • Demonstrate a pipeline failure from a deliberately introduced issue and a clean run that promotes the model
• Module 13: AI Infrastructure, Access Control & Supply Chain Security
  • AI infrastructure components: compute/GPU workloads, containers, model-serving endpoints, vector database storage
  • Identity and Access Management (IAM) for AI infrastructure: RBAC and least privilege for dataset modification, model deployment, and registry/tuning-parameter access
  • Environment hardening: network segmentation, GPU cluster isolation, and lateral-movement prevention
  • Container security for AI workloads
  • Open-source model and public dataset supply chain risk, third-party AI API risk
  • Model provenance verification
  • SBOM and AI-BOM concept: what they capture and why they matter for AI supply chain transparency
  • Practical Exercises:
    • Deploy an LLM gateway(LiteLLM) in front of the application, configured with rate limits, authentication, and request logging
    • Scan a container image for vulnerabilities(Trivy)
    • Using Model Provenance Toolkit
    • Generate an SBOM/AI-BOM for a sample AI app using a guided template (Syft + CycloneDX)
• Module 14: AI Security Monitoring and Incident Response
  • ML drift monitoring: data drift vs. concept drift, reviewing an Evidently AI drift report
  • AI security monitoring overview: prompt, API usage, model query, data access, tool execution, and agent action monitoring
  • Output validation: monitoring deployed outputs for sensitive data leakage, copyright infringement, or malicious content generation
  • LLM observability tooling: Arize Phoenix and Langfuse/Langsmith for tracing calls, retrieval steps, tool invocations
  • Abuse detection: spotting attacker behaviour over time (high-volume querying, repeated extraction attempts, repeated jailbreak attempts)
  • Vulnerability intake, classification, and severity rating for AI issues
  • AI incident scenarios: prompt injection, sensitive data exposure, model extraction, poisoned dataset, unauthorized model replacement, API abuse, shadow AI
  • Incident response playbooks: quarantine, retrain, and rollback procedures for compromised models
  • Honeypots in AI Applications
  • Practical Exercises:
    • Review a pre-built Evidently AI drift report and configure a basic alert threshold
    • Analyze AI API abuse logs for extraction/jailbreak patterns
    • Building and deploying a honeypot in RAG Pipeline
• Module 15: AI Governance and Responsible AI Practice
  • NIST AI RMF mapping exercise (Govern, Map, Measure, Manage)
  • Responsible AI properties of controls built in the course
  • Secure AI SDLC: Acquire vs Build governance questions
  • AI-BOM’s role in governance and audit evidence
  • Practical Exercises:
    • Given a sample AI system(a built combination of one ML model and one LLM/RAG application from earlier in the course), learners:
      • Review an AI API for missing authentication/authorization (Postman/Burp Suite)
      • Identify broken access control in a RAG scenario
      • Test for API key exposure and weak token handling

• Security professionals moving into AI:
  • Security Engineers
  • Security Architects
  • Penetration Testers/Red Teamers
  • SOC Analysts
  • AppSec Engineers
  • DevSecOps Engineers
• AI and engineering professionals moving into security:
  • Data Scientists
  • ML Engineers
  • LLM Engineers
  • MLOps/Platform Engineers

• Foundational AI and cybersecurity knowledge. Preferably should have done Infosectrain’s AI Cybersecurity Foundation Program or should have equivalent knowledge.
• Comfortable working with the command line, configuration files, and gen-AI assisted coding.

After completion of this course, you will be able to:

• Build a security-first mental model of AI systems: attack surface, trust boundaries, and defense-in-depth.
• Build, attack, and secure both traditional ML/DL security controls and GenAI/LLM/agentic applications.
• Threat-model AI systems using STRIDE, MITRE ATLAS, OWASP ML/LLM/Agentic Top 10, and MAESTRO.
• Apply adversarial techniques like poisoning, evasion, extraction, prompt injection, jailbreaking, memory/tool poisoning against systems built in the course.
• Implement defense-in-depth across data, model, application, and infrastructure layers: guardrails, gateways, adversarial training, artifact integrity, access control, and supply chain checks.
• Secure the full MLOps/LLMOps pipeline: CI/CD security gates, dataset/prompt versioning, promotion gates, and rollback procedures.
• Monitor AI systems in production and respond to AI-specific incidents like drift, abuse, extraction, prompt injection, shadow AI.
• Map technical controls to NIST AI RMF and Responsible AI properties, and produce governance evidence a real audit or compliance function would ask for.