Enterprise AI/ML Checklist
A comprehensive checklist for developing and deploying enterprise-grade AI/ML solutions,
focusing on data management, model development, MLOps, monitoring, and ethical
considerations. This checklist covers essential machine learning practices while embracing
modern AI/ML architectures and methodologies. It spans both classical/predictive ML systems
(the data pipelines, model training, drift detection, and fairness metrics below) and
generative AI/LLM systems (prompt injection defenses, RAG quality, agentic AI governance,
and AI Act/NIST/ISO risk tiering) — enterprises now run both, and neither should be governed
by the other's playbook alone.
Model Development
Required
Model Architecture
ⓘ
Implementation Questions:
Why was this specific model architecture chosen for the problem domain?
What alternative architectures were considered and why were they rejected?
How do model complexity and interpretability requirements balance?
What are the computational requirements and performance characteristics?
How does the architecture handle different input sizes and data types?
What are the model's scalability limitations and bottlenecks?
Key Considerations:
Document architectural decision records (ADRs) with rationale
Consider model inference latency and throughput requirements
Plan for model updates and architecture evolution
Validate architecture against business and technical constraints
Red Flags:
Using complex models without justification for simple problems
No documentation of why specific architecture was chosen
Architecture that doesn't align with deployment constraints
Over-engineering or under-engineering for the use case
Training Pipeline
ⓘ
Implementation Questions:
What steps ensure exact reproducibility of training runs (seed management,
environment isolation)?
How do you handle hyperparameter configuration and experimentation?
What compute resource management and scaling strategies are implemented?
How do you track and log training progress, metrics, and intermediate
results?
What failure recovery and checkpointing mechanisms exist?
How do you handle distributed training across multiple GPUs or nodes?
Key Considerations:
Containerize training environments with fixed dependency versions
Implement automated hyperparameter tracking and experiment logging
Use configuration files to manage training parameters
Implement early stopping and model checkpointing strategies
Red Flags:
Training runs that cannot be reproduced due to missing configuration
Manual training processes prone to human error
No monitoring of training resource usage and costs
Training failures that lose hours or days of computation
Model Validation
ⓘ
Implementation Questions:
What validation techniques are used (cross-validation, holdout, time-series
splits)?
How do you ensure validation data is representative of production data?
What metrics are tracked beyond accuracy (precision, recall, F1, AUC)?
How do you validate model performance across different data segments?
What statistical tests are used to assess model significance?
How do you validate model robustness and edge case handling?
Key Considerations:
Use stratified sampling to maintain class balance in validation sets
Implement business-relevant metrics alongside technical metrics
Validate model performance across different time periods and conditions
Include bias and fairness assessments in validation procedures
Red Flags:
Data leakage between training and validation sets
Validation only on clean, perfect data that doesn't reflect reality
Over-reliance on single metrics without considering business impact
No validation of model behavior on edge cases or adversarial inputs
Model Versioning
ⓘ
Implementation Questions:
How do you version and track model artifacts (weights, architecture,
hyperparameters)?
What metadata is stored with each model version (performance metrics,
training data, timestamp)?
How do you handle model promotion through development, staging, and
production environments?
What rollback procedures exist when new model versions underperform?
How do you manage model dependencies and environment compatibility?
What tools do you use for model registry and lifecycle management?
Key Considerations:
Use semantic versioning with clear promotion criteria between versions
Store complete model lineage including data versions and code commits
Implement automated model validation before version promotion
Maintain model performance benchmarks across versions
Red Flags:
No systematic approach to model versioning and tracking
Production models that cannot be traced to specific training runs
Model artifacts stored without proper metadata or lineage
No rollback strategy when models degrade in production
MLOps
Required
CI/CD Pipeline
ⓘ
Implementation Questions:
What ML-specific tests are automated (data validation, model testing,
performance regression)?
How do you handle model artifacts in your CI/CD pipeline?
What triggers automated retraining and model updates?
How do you implement canary deployments and A/B testing for models?
What rollback mechanisms exist when model deployments fail?
How do you manage environment consistency across development, staging, and
production?
Key Considerations:
Implement automated data quality checks and schema validation
Include model performance benchmarks in your test suite
Use containerization for consistent ML environment deployment
Implement gradual rollout strategies with performance monitoring
Red Flags:
Manual model deployment processes prone to human error
No automated testing of model performance or data quality
All-or-nothing deployments without gradual rollout capabilities
Environment inconsistencies between development and production
Model Registry
ⓘ
Implementation Questions:
What information is tracked for each model in the registry (version,
performance, metadata)?
How do you manage model promotion through different environments?
What approval workflows exist for model deployment to production?
How do you track which model versions are deployed where?
What lineage information connects models to training data and code?
How do you handle model deprecation and retirement?
Key Considerations:
Implement automated model registration from training pipelines
Store comprehensive model metadata including performance benchmarks
Enable model comparison and diff capabilities
Integrate with deployment tools for consistent model management
Red Flags:
Models deployed without proper registry tracking
No clear promotion criteria between registry stages
Missing model metadata that prevents effective comparison
Registry that becomes disconnected from actual deployments
Model Serving
ⓘ
Implementation Questions:
What model serving framework supports your latency and throughput
requirements?
How do you handle model loading, caching, and memory management?
What auto-scaling policies respond to prediction request volume?
How do you implement batch vs. real-time serving strategies?
What monitoring exists for model serving performance and errors?
How do you handle model versioning and A/B testing in serving?
Key Considerations:
Choose serving solutions that match your latency requirements (milliseconds
vs. seconds)
Implement model caching and pre-loading strategies
Use containerized serving with orchestration for scalability
Monitor resource utilization and cost optimization opportunities
Red Flags:
Serving infrastructure that cannot handle peak traffic loads
No monitoring of model serving latency and availability
Models that require manual deployment and scaling
Serving costs that are disproportionate to business value
Monitoring Setup
ⓘ
Implementation Questions:
What model performance metrics are monitored in real-time?
How do you detect and alert on data drift and model degradation?
What infrastructure metrics track serving performance and availability?
How do you correlate model performance with business outcomes?
What dashboards provide visibility into model health?
How do you handle alert fatigue while maintaining responsiveness?
Key Considerations:
Monitor both technical metrics (latency, error rates) and business metrics
(accuracy, precision)
Implement statistical process control for drift detection
Create tiered alerting based on severity and impact
Provide self-service dashboards for stakeholders
Red Flags:
Model degradation discovered weeks after it begins
No correlation between model metrics and business impact
Alert storms that overwhelm the team
Manual monitoring processes that create delays
Model Monitoring
Suggested
Automated Retraining
ⓘ
Implementation Questions:
What conditions trigger automated retraining (performance degradation, data
drift, time intervals)?
How do you ensure retraining uses appropriate data windows and validation
procedures?
What quality gates and approval processes govern automated model deployment?
How do you handle retraining failures and fallback to previous model
versions?
What resource management ensures retraining doesn't interfere with
production workloads?
How do you track and compare the performance of automatically retrained
models?
Key Considerations:
Implement gradual rollout strategies for automatically retrained models
Use comprehensive validation pipelines before deploying retrained models
Monitor business metrics, not just technical metrics, when evaluating
retraining success
Maintain human oversight and intervention capabilities in automated
workflows
Red Flags:
Automated retraining that deploys models without adequate validation
Retraining triggers that fire too frequently, causing model instability
No fallback mechanism when automated retraining produces worse models
Resource competition between retraining and production inference workloads
Performance Dashboard
ⓘ
Implementation Questions:
What key performance indicators are displayed for different stakeholder
groups (technical teams, business users)?
How do you visualize trends, anomalies, and comparisons across multiple
models and time periods?
What drill-down capabilities allow users to investigate performance issues
or understand model behavior?
How do you ensure dashboards update in real-time and handle high-volume
metrics efficiently?
What customization options allow different teams to focus on relevant
metrics for their roles?
How do you integrate model performance with business outcome metrics in
unified views?
Key Considerations:
Design role-based dashboards that show relevant metrics for each audience
Include both technical metrics (accuracy, latency) and business metrics
(revenue impact, user satisfaction)
Provide contextual information and alerts that help users understand when
action is needed
Enable easy sharing and collaboration features for dashboard insights
Red Flags:
Dashboards that show data without context or actionable insights
Performance visualizations that are too technical for business stakeholders
Dashboards that become outdated or show stale data during critical periods
No clear connection between model performance metrics and business value
delivered
Ethics & Fairness
Required
Bias Detection
ⓘ
Implementation Questions:
What protected attributes and sensitive variables are monitored for bias?
How do you measure and quantify different types of bias (statistical,
individual, group)?
What bias detection tools and techniques are integrated into your pipeline?
How do you balance fairness constraints with model performance?
What is your process for addressing bias when it's detected?
How do you validate bias mitigation effectiveness over time?
Key Considerations:
Implement multiple fairness metrics (demographic parity, equalized odds,
individual fairness)
Use techniques like adversarial debiasing, fairness constraints, or
post-processing
Document bias assessment results and mitigation strategies
Involve diverse stakeholders in defining fairness requirements
Red Flags:
No systematic bias testing across different demographic groups
Relying solely on overall accuracy metrics without fairness assessment
Bias discovered only after negative real-world impact
No clear process for handling fairness-performance trade-offs
Fairness Metrics
ⓘ
Implementation Questions:
Which fairness metrics are most appropriate for your use case (demographic
parity, equalized odds, equalized opportunity)?
How do you define and measure fairness across different protected attributes
(race, gender, age, disability)?
What thresholds and acceptable ranges are established for each fairness
metric?
How do you handle intersectionality when multiple protected attributes are
involved?
What trade-offs exist between different fairness metrics and overall model
performance?
How frequently are fairness metrics evaluated and reported to stakeholders?
Key Considerations:
Implement multiple fairness metrics as no single metric captures all aspects
of fairness
Establish fairness thresholds in collaboration with domain experts and
affected communities
Create fairness dashboards and regular reporting mechanisms for stakeholders
Document the rationale behind chosen fairness metrics and thresholds
Red Flags:
Relying on a single fairness metric without considering others
Setting fairness thresholds without input from affected communities or
stakeholders
Fairness metrics that are measured only during development but not in
production
No clear documentation of fairness requirements and trade-offs
Model Explainability
ⓘ
Implementation Questions:
What explainability techniques are appropriate for your model type (LIME,
SHAP, attention maps, feature importance)?
How do you provide explanations at different levels (global, local,
counterfactual)?
What is the target audience for explanations (data scientists, business
users, end users, regulators)?
How do you validate that explanations are accurate and meaningful?
What processes ensure explanations remain consistent as models are updated?
How do you balance explainability requirements with model performance and
complexity?
Key Considerations:
Implement multiple explanation techniques to provide comprehensive
interpretability
Tailor explanation formats and complexity to different stakeholder needs
Validate explanations against domain expert knowledge and intuition
Integrate explainability tools into model development and deployment
workflows
Red Flags:
Using black-box models in high-stakes decisions without adequate
explainability
Explanations that are technically correct but incomprehensible to intended
users
No validation of explanation accuracy or faithfulness to actual model
behavior
Explanations that become outdated when models are retrained or updated
Ethics Guidelines
ⓘ
Implementation Questions:
What ethical principles and values guide your AI development (transparency,
accountability, human oversight)?
How do you translate high-level ethical principles into specific, actionable
guidelines for teams?
What review processes ensure AI projects adhere to ethical guidelines
throughout development?
How do you handle ethical dilemmas and conflicts between different ethical
principles?
What training and education programs ensure team members understand and
apply ethical guidelines?
How do you update and evolve ethical guidelines as AI technology and
societal norms change?
Key Considerations:
Develop clear, specific guidelines that address common ethical issues in AI
development
Implement mandatory ethics reviews at key project milestones
Provide practical tools and checklists to help teams apply ethical
principles
Create escalation procedures for complex ethical decisions and conflicts
Red Flags:
Vague ethical statements without specific implementation guidance
No formal process for reviewing ethical implications of AI projects
Ethics considered only at the end of development rather than throughout
No mechanisms for updating guidelines as ethical understanding evolves
Generative AI & LLM
Required
Prompt Injection & Jailbreak Defense
ⓘ
Implementation Questions:
What defenses are in place against direct and indirect prompt injection
(e.g., malicious instructions embedded in retrieved documents, web content,
or user uploads)?
How do you distinguish trusted system/developer instructions from untrusted
user or tool-retrieved content architecturally?
What jailbreak detection techniques identify attempts to bypass safety
guardrails (role-play framing, encoding tricks, multi-turn escalation)?
How do you test for prompt injection vulnerabilities before and after
deployment?
What happens when an injection or jailbreak attempt is detected — is it
logged, blocked, and escalated?
How do you handle prompt injection risks introduced through third-party
plugins, tools, or agent integrations?
Key Considerations:
Use input/output filtering and instruction-hierarchy techniques (system
prompts privileged over retrieved/user content)
Sandbox and apply least-privilege to any tool or plugin the LLM can invoke
Continuously red-team with updated jailbreak corpora rather than a one-time
pre-launch test
Log prompts, retrieved context, and model outputs for forensic review of
suspected injection attempts
Red Flags:
System prompts or credentials that leak through crafted user inputs
No distinction in the model pipeline between trusted instructions and
untrusted retrieved/user content
Jailbreak testing performed once at launch and never revisited as new
techniques emerge
LLM-driven tools that execute actions directly from unsanitized model output
RAG Evaluation & Retrieval Quality
ⓘ
Implementation Questions:
What metrics measure retrieval quality (recall@k, precision, context
relevance, groundedness)?
How do you evaluate whether generated answers are actually grounded in
retrieved context versus fabricated?
What process re-indexes or refreshes the knowledge base, and how do you
prevent stale or conflicting sources from degrading answers?
How do you test RAG pipelines against adversarial or out-of-distribution
queries?
What chunking, embedding, and re-ranking strategies were evaluated and why
were they chosen?
How do you measure end-to-end RAG quality (not just retrieval, but the final
answer) before and after pipeline changes?
Key Considerations:
Maintain a labeled evaluation set of query/expected-context/expected-answer
triples for regression testing
Track groundedness/faithfulness scores alongside traditional retrieval
metrics
Version and audit the knowledge corpus the same way you version training
data
Include citation/source attribution in generated answers where feasible
Red Flags:
No automated evaluation harness for retrieval or answer quality; changes
assessed only by spot-checking
Retrieved context that is stale, duplicated, or contradictory without
detection
Generated answers that cite sources they did not actually retrieve
RAG pipeline changes deployed without regression testing against a held-out
evaluation set
Vector Database Access Control
ⓘ
Implementation Questions:
What authentication and authorization controls restrict who and what
services can query or write to the vector store?
How do you enforce document-level or tenant-level access control so
retrieval never surfaces content a requesting user isn't entitled to see?
How is embedded data (which may contain PII or sensitive business data)
encrypted at rest and in transit?
What audit logging exists for vector database queries, especially bulk or
anomalous access patterns?
How do you handle data deletion and "right to be forgotten" for embeddings,
not just source documents?
How do you isolate vector indexes across environments (dev/staging/prod) and
across customers in multi-tenant deployments?
Key Considerations:
Apply the same access-control rigor to embeddings/vector stores as to the
source data they were derived from
Filter retrieval results by the requesting user's permissions before they
ever reach the prompt context
Encrypt embeddings and metadata, and rotate credentials for vector database
access
Treat embedding deletion as part of the standard data-retention/deletion
workflow
Red Flags:
A shared vector index across tenants or users with no row/namespace-level
filtering
Embeddings that retain sensitive data after the source document was deleted
Vector database endpoints reachable without authentication or with overly
broad service credentials
No audit trail of who queried what, making leak investigation impossible
Hallucination Detection & Mitigation
ⓘ
Implementation Questions:
What techniques detect factually incorrect or unsupported model outputs
(self-consistency checks, retrieval grounding, claim verification)?
How do you measure hallucination rate for your specific use case, and what
threshold is acceptable?
What guardrails prevent hallucinated outputs from reaching end users
unreviewed in high-stakes workflows?
How do you handle model confidence signaling — does the system communicate
uncertainty to users?
What feedback loop captures user-reported hallucinations and feeds them back
into evaluation or fine-tuning?
How do you test for hallucination under adversarial or out-of-scope prompts
specifically designed to induce fabrication?
Key Considerations:
Ground generation in retrieved, citable sources wherever possible rather
than relying on parametric knowledge
Use automated fact-checking or claim-verification models as a secondary
check on high-stakes outputs
Require human review for hallucination-sensitive domains (legal, medical,
financial advice)
Track hallucination rate as a first-class production metric, not just
accuracy
Red Flags:
No systematic measurement of hallucination rate before or after deployment
Fabricated citations, statistics, or facts presented with the same
confidence as verified content
Users given no indication of model uncertainty or the possibility of
fabricated content
Hallucination incidents discovered only through customer complaints rather
than proactive monitoring
Suggested
Agentic AI Guardrails
ⓘ
Implementation Questions:
What tools, APIs, or systems can AI agents invoke, and how is that
permission set scoped per agent and per task?
How do you enforce least-privilege so an agent can only take actions
necessary for its assigned task?
What guardrails prevent an agent from taking irreversible or high-impact
actions (financial transactions, data deletion, external communications)
without approval?
How do you constrain multi-step autonomous planning/execution loops to
prevent runaway behavior or excessive resource consumption?
What sandboxing or simulation environment is used to test agent behavior
before granting production tool access?
How do you audit and replay an agent's full chain of tool calls and
decisions after an incident?
Key Considerations:
Define an explicit allowlist of tools/actions per agent role rather than
granting broad system access
Require approval gates or rate limits for actions above a defined
risk/impact threshold
Log every tool call, input, and output in the agent's action trace for
auditability
Test agents in a sandboxed environment with synthetic tools before
connecting to production systems
Red Flags:
Agents granted broad API keys or credentials far beyond what their task
requires
No approval step before an agent executes irreversible or high-cost actions
Agent action logs that are incomplete or not correlated with the reasoning
that triggered them
Multi-agent or multi-step workflows with no cap on iterations, cost, or
elapsed time
Human-in-the-Loop Checkpoints
ⓘ
Implementation Questions:
For which agent actions or generative outputs is human review mandatory
before execution or publication?
How do you determine the risk threshold that triggers a human checkpoint
versus fully autonomous operation?
What interface do reviewers use to approve, reject, or modify an agent's
proposed action?
How do you prevent checkpoint fatigue where reviewers rubber-stamp requests
without meaningful review?
What happens when a human reviewer is unavailable — does the agent fail safe
(block) or proceed?
How do you track override rates and reviewer decisions to improve agent
behavior and checkpoint placement over time?
Key Considerations:
Tier checkpoints by risk: low-risk actions proceed autonomously, high-risk
actions require explicit human sign-off
Design review interfaces that surface the agent's reasoning and evidence,
not just the final action
Default to a safe, blocked state when human review is required but
unavailable
Periodically audit override/approval patterns to recalibrate where
checkpoints are placed
Red Flags:
"Human-in-the-loop" that exists on paper but reviewers approve everything
without real scrutiny
No defined risk tiering — either everything requires review (unscalable) or
nothing does (unsafe)
Agents that proceed autonomously by default when a reviewer fails to respond
No audit trail of what a human approved versus what the agent originally
proposed
AI Risk Tiering & Regulatory Compliance
Required
EU AI Act Risk Tiering
ⓘ
Implementation Questions:
Has each AI system been classified into an EU AI Act risk tier
(unacceptable/prohibited, high-risk, limited-risk, minimal-risk)?
For high-risk systems, what conformity assessment, risk management system,
and human oversight measures are documented?
For general-purpose AI models (GPAI), what transparency and systemic-risk
documentation is maintained (technical documentation, training data summary,
copyright compliance)?
For limited-risk systems (e.g., chatbots, deepfake generators), what
transparency obligations (disclosing AI interaction, labeling synthetic
content) are implemented?
How do you re-assess risk tier classification when a system's use case,
deployment context, or capabilities change?
Who owns the AI Act compliance process, and how is it kept current as
enforcement phases take effect (prohibited practices since Feb 2025, GPAI
since Aug 2025, full applicability Aug 2026)?
Key Considerations:
Maintain a system-level inventory that records each AI system's risk tier
classification and supporting rationale
Build conformity assessment and CE-marking readiness into the development
lifecycle for high-risk systems, not as a late-stage gate
Track GPAI obligations separately, since foundation models you build on may
carry their own upstream compliance requirements
Reassess tiering whenever a system is repurposed or deployed into a new
context, since risk tier is use-case dependent, not model-dependent
Red Flags:
No documented risk-tier classification for any deployed AI system
High-risk systems in production without a risk management file, human
oversight design, or conformity assessment
Chatbots or generative features that don't disclose to users that they're
interacting with AI
Compliance treated as a one-time exercise rather than revisited as
regulatory phases and system use cases evolve
NIST AI RMF Alignment
ⓘ
Implementation Questions:
How does your AI governance program map to the four NIST AI RMF functions
(Govern, Map, Measure, Manage)?
What organizational structures and policies implement the "Govern" function
(accountability, culture, risk tolerance)?
How do you "Map" context and risks for each AI system before development
begins?
What quantitative and qualitative methods do you use to "Measure" identified
risks (bias, robustness, safety, security)?
How are measured risks prioritized and acted upon under the "Manage"
function, and who has authority to stop or roll back a deployment?
How do you use the NIST Generative AI Profile to address risks specific to
LLMs and generative systems?
Key Considerations:
Treat AI RMF as a continuous lifecycle process, not a one-time checklist —
Govern/Map/Measure/Manage repeat as systems evolve
Align existing model risk management and MLOps processes to the four
functions rather than building a parallel process
Use the NIST Generative AI Profile as a supplement covering genAI-specific
risks (confabulation, data privacy, content provenance)
Document risk decisions and residual risk acceptance for audit and
regulatory readiness
Red Flags:
AI risk management that exists only as documentation with no operational
tie-in to development or deployment gates
No designated accountable owner for AI risk decisions (the "Govern"
function)
Risks identified during "Map" that are never revisited during "Measure" or
"Manage"
Generative AI systems evaluated using only classical-ML risk criteria,
ignoring genAI-specific failure modes
ISO/IEC 42001 AI Management System
ⓘ
Implementation Questions:
Does your organization have a documented AI Management System (AIMS) aligned
to ISO/IEC 42001, or a roadmap toward certification?
What AI policy, objectives, and top-management commitment are documented as
required by the standard?
How do you conduct and document AI-specific risk and impact assessments as
part of the AIMS?
What internal audit and management review cadence exists for the AI
management system?
How is the AIMS scoped — which AI systems, business units, and third-party
AI dependencies are in scope?
How does ISO/IEC 42001 certification (or alignment) get evidenced to
customers, auditors, or regulators as part of due diligence?
Key Considerations:
Scope the AIMS deliberately — start with highest-risk or highest-visibility
AI systems rather than boiling the ocean
Integrate AIMS processes with existing ISO 27001/SOC 2 management systems
where controls overlap
Assign clear ownership for the AI policy, risk register, and continual
improvement cycle
Use ISO/IEC 42001 as the operational management-system layer that
complements EU AI Act and NIST AI RMF technical/regulatory requirements
Red Flags:
AI policy that exists as a static document with no management review or
continual improvement process
No AI-specific risk register distinct from general IT/security risk
registers
AIMS scope that excludes third-party or vendor-supplied AI/LLM components
actually in production use
Certification or alignment claimed without evidence of internal audits or
documented corrective actions
Documentation
Required
Model Cards
ⓘ
Implementation Questions:
What model performance metrics, limitations, and intended use cases are
documented in each model card?
How do you document training data characteristics, biases, and
representativeness?
What evaluation datasets and fairness assessments are included?
How do you document model architecture decisions and hyperparameter choices?
What ethical considerations and potential risks are outlined?
How often are model cards updated as models evolve or new issues are
discovered?
Key Considerations:
Follow standardized model card templates (Google Model Cards, Hugging Face
Cards)
Include quantitative performance metrics across different demographic groups
Document known limitations, failure modes, and out-of-scope use cases
Make model cards accessible to both technical and non-technical stakeholders
Red Flags:
Model cards that are created once and never updated
Missing documentation of model limitations or potential biases
Technical jargon that makes cards inaccessible to business stakeholders
No process for reviewing and approving model card content
Foundation Model / System Cards
ⓘ
Implementation Questions:
How do you document training data provenance at a high level (data sources,
collection methods, cutoff dates) without exposing proprietary or sensitive
details?
What intended uses and explicitly out-of-scope/misuse cases are documented
for the foundation model or LLM-based system?
What known limitations are disclosed (context window constraints,
multilingual performance gaps, reasoning failure modes, knowledge cutoff)?
What safety evaluation results are documented (red-teaming summaries,
harmful-content benchmarks, bias evaluations specific to generative
outputs)?
How do you document the model's content provenance and watermarking/labeling
approach for generated content, where applicable?
How do you handle documentation when building on top of a third-party
foundation model you didn't train yourself (vendor model/system cards)?
Key Considerations:
Distinguish a "system card" (documenting your deployed application,
including prompts, guardrails, and RAG/tool integrations) from a raw
foundation-model card
Publish high-level training data provenance and licensing posture even when
full dataset details can't be disclosed
Include safety evaluation summaries (jailbreak resistance, harmful content
refusal rates) alongside traditional performance metrics
Update the system card whenever the underlying model, prompt, or tool
integrations change materially
Red Flags:
Treating a vendor's foundation-model card as sufficient documentation for
your own deployed system, prompts, and integrations
No disclosure of intended use / misuse boundaries, leaving users to discover
limitations through failure
Safety evaluation section that is copied once at launch and never updated as
the model or its integrations change
No documentation of what happens when the underlying third-party model is
silently updated by the vendor
Data Documentation
ⓘ
Implementation Questions:
What metadata is captured for each data source (origin, freshness, quality,
ownership)?
How do you document data schema evolution and backward compatibility?
What transformations, feature engineering, and preprocessing steps are
documented?
How do you track data lineage from raw sources to final features?
What data quality issues and remediation steps are documented?
How do you document data sampling strategies and potential biases?
Key Considerations:
Create data dictionaries with business-friendly descriptions
Document data collection methodologies and potential sampling biases
Maintain version-controlled schema documentation with change logs
Include data quality metrics and acceptable thresholds
Red Flags:
Undocumented data transformations that affect model reproducibility
Missing documentation of data collection biases or limitations
Outdated schema documentation that doesn't reflect current data structure
No clear ownership or contact information for data sources
Pipeline Documentation
ⓘ
Implementation Questions:
What pipeline architecture diagrams show data flow and component
interactions?
How do you document pipeline dependencies, configurations, and environment
requirements?
What deployment procedures, rollback processes, and monitoring setups are
documented?
How do you document pipeline failure modes and recovery procedures?
What performance characteristics and resource requirements are specified?
How do you maintain runbooks for pipeline operations and troubleshooting?
Key Considerations:
Create visual pipeline diagrams with clear data flow and decision points
Document all configuration parameters and environment variables
Maintain step-by-step deployment and rollback procedures
Include performance benchmarks and resource utilization guidelines
Red Flags:
Undocumented pipeline components that create knowledge silos
Missing documentation of failure recovery procedures
Outdated deployment documentation that doesn't reflect current processes
No documentation of pipeline performance characteristics or SLAs
API Documentation
ⓘ
Implementation Questions:
What API endpoints are documented with request/response schemas and
examples?
How do you document authentication, authorization, and rate limiting
requirements?
What error codes, status messages, and troubleshooting guidance are
provided?
How do you document API versioning, deprecation policies, and migration
paths?
What performance characteristics, latency expectations, and SLAs are
specified?
How do you provide interactive documentation and code examples for different
languages?
Key Considerations:
Use OpenAPI/Swagger specifications for standardized, interactive
documentation
Provide realistic examples with actual data formats and edge cases
Document all possible error scenarios with appropriate response codes
Include SDK examples and client libraries for popular programming languages
Red Flags:
API documentation that becomes stale and doesn't reflect actual endpoints
Missing documentation of error handling and edge cases
No examples or code snippets for common integration patterns
Unclear authentication requirements or API usage policies
Suggested
Decision Records
ⓘ
Implementation Questions:
What architectural decisions are captured in decision records (model
selection, infrastructure choices, design patterns)?
How do you document the context, alternatives considered, and rationale for
each decision?
What is the process for creating, reviewing, and approving decision records?
How do you track the consequences and outcomes of architectural decisions
over time?
What template or format ensures consistent decision record quality?
How do you make decision records discoverable and searchable for future
reference?
Key Considerations:
Use standardized ADR templates with status, context, decision, and
consequences
Document both technical and business rationale for decisions
Review decisions periodically to assess outcomes and learn from results
Make decision records part of the code repository for version control
Red Flags:
Major architectural decisions made without documentation or rationale
Decision records that are never revisited or updated with outcomes
No consistent format or process for creating decision records
Important decisions buried in meeting notes or email threads
Experiment Tracking
ⓘ
Implementation Questions:
What experiment metadata is tracked (hypothesis, parameters, datasets,
results)?
How do you document experiment setup, configuration, and reproducibility
requirements?
What tools are used for experiment logging and result comparison (MLflow,
Weights & Biases)?
How do you track negative results and failed experiments for future
reference?
What process exists for peer review and validation of experiment results?
How do you organize and search experiments across different projects and
teams?
Key Considerations:
Implement automated experiment logging with version control integration
Document clear hypotheses and success criteria before running experiments
Track resource usage and computational costs for experiment planning
Create experiment summaries that highlight key insights and learnings
Red Flags:
Experiments that cannot be reproduced due to missing documentation
No systematic tracking of negative results or failed approaches
Experiment logs that are scattered across different tools and platforms
Missing documentation of experiment rationale and business context
Security & Compliance
Suggested
Attack Prevention
ⓘ
Implementation Questions:
What types of adversarial attacks are you defending against (evasion,
poisoning, extraction, inference)?
How do you implement adversarial training and robust optimization
techniques?
What input validation and sanitization measures prevent malicious inputs?
How do you detect and respond to ongoing adversarial attacks in production?
What rate limiting and behavioral analysis protect against systematic
attacks?
How do you balance robustness measures with model performance and usability?
Key Considerations:
Implement gradient masking and defensive distillation techniques
Deploy anomaly detection for unusual query patterns and attack signatures
Use ensemble methods and randomized defenses to increase attack difficulty
Establish incident response procedures for confirmed adversarial attacks
Red Flags:
No defense mechanisms against known adversarial attack vectors
Attack prevention measures that are easily circumvented or bypassed
Lack of monitoring for adversarial attack attempts and patterns
Defensive measures that significantly degrade legitimate user experience
Adversarial Robustness Testing
ⓘ
Implementation Questions:
What adversarial attack methods are used to test model robustness (FGSM,
PGD, C&W)?
How do you systematically evaluate model vulnerability across different
attack scenarios?
What robustness metrics and benchmarks do you use to measure defense
effectiveness?
How do you test robustness against both white-box and black-box attacks?
What is your process for addressing vulnerabilities discovered during
robustness testing?
How often do you conduct robustness testing and update defenses?
Key Considerations:
Implement automated adversarial testing in your CI/CD pipeline
Use diverse attack libraries and frameworks (CleverHans, Foolbox, ART)
Test robustness across different input domains and attack strengths
Establish robustness thresholds and acceptance criteria for deployment
Red Flags:
Models deployed without any adversarial robustness testing
Testing only against basic attacks while ignoring advanced techniques
No systematic approach to fix vulnerabilities found during testing
Robustness testing performed only once without regular updates
LLM Red-Teaming & Safety Evaluations
ⓘ
Implementation Questions:
What red-teaming process specifically targets LLM/generative behaviors
(jailbreaks, harmful content generation, toxic or biased outputs, PII
leakage through generation)?
How do you differentiate this from classical adversarial-ML testing
(evasion/poisoning attacks), which targets model inputs/training rather than
generated outputs?
What harm categories are in scope for red-teaming (violence, self-harm,
illegal activity, misinformation, IP infringement, privacy leakage)?
Do you use internal red teams, external/third-party red teams, or automated
adversarial-prompt generation — or a combination?
How are red-team findings triaged, remediated, and re-tested before and
after each model or prompt-pipeline update?
What ongoing safety evaluation cadence exists in production, beyond a
one-time pre-launch red-team exercise?
Key Considerations:
Maintain a living library of known jailbreak techniques and harmful-prompt
categories, refreshed as new techniques are published
Combine manual red-teaming (creative, adversarial human testers) with
automated adversarial-prompt generation for coverage and scale
Track safety eval results as regression tests re-run on every model, prompt,
or RAG-pipeline change
Report red-team findings and remediation status to the same governance body
reviewing classical-ML fairness/bias results
Red Flags:
Red-teaming performed once before launch with no re-testing after prompt,
model, or tool changes
Red-team scope that only covers classical adversarial-ML attacks and omits
jailbreak/harmful-generation testing
Findings logged but never fed back into prompt hardening, guardrail updates,
or model selection decisions
No safety evaluation coverage for a newly added tool/agent capability before
it goes live
Model Extraction Protection
ⓘ
Implementation Questions:
What monitoring systems detect suspicious query patterns indicative of model
extraction attempts?
How do you implement query rate limiting and request throttling to prevent
extraction?
What techniques protect against model stealing and reverse engineering
(output perturbation, query budgets)?
How do you detect and respond to systematic evasion attacks against your
models?
What logging and analytics track potential intellectual property theft
attempts?
How do you balance extraction protection with legitimate high-volume usage?
Key Considerations:
Implement ML-based anomaly detection for unusual query patterns
Use output randomization and noise injection as extraction countermeasures
Deploy honeypot queries and detection mechanisms for automated extraction
tools
Establish legal frameworks and terms of service to deter extraction attempts
Red Flags:
No monitoring for systematic querying or suspicious usage patterns
Model APIs that allow unlimited queries without rate limiting
High-confidence predictions that make extraction easier for attackers
No incident response plan for confirmed model extraction attempts
Differential Privacy
ⓘ
Implementation Questions:
What differential privacy mechanisms and epsilon values are appropriate for
your sensitivity requirements?
How do you implement federated learning to keep sensitive data distributed
while enabling collaborative model training?
What privacy accounting and budget management systems track cumulative
privacy loss?
How do you validate that differential privacy provides meaningful protection
without excessive utility loss?
What secure aggregation protocols protect individual contributions in
federated learning?
How do you handle client dropout and byzantine behavior in federated
learning systems?
Key Considerations:
Implement both local and global differential privacy depending on trust
models
Use advanced composition theorems to optimize privacy budgets across
multiple queries
Deploy secure multi-party computation for privacy-preserving aggregation
Establish governance frameworks for privacy parameter selection and review
Red Flags:
Differential privacy parameters chosen without formal privacy analysis
Federated learning systems vulnerable to inference attacks on model updates
No monitoring of cumulative privacy budget depletion over time
Privacy mechanisms that provide theoretical but not practical protection
AI/ML Security Testing in CI/CD
ⓘ
Implementation Questions:
What AI/ML-specific security tests are automated in your CI/CD pipelines
(adversarial testing, data poisoning detection)?
How do you conduct regular penetration testing specifically targeting ML
systems and APIs?
What security gates and quality checks prevent vulnerable models from
reaching production?
How do you test for security regressions when updating models or ML
infrastructure?
What threat modeling processes identify potential attack vectors in your ML
systems?
How do you integrate ML security testing with traditional application
security testing?
Key Considerations:
Implement automated adversarial example generation and robustness testing
Deploy ML-specific vulnerability scanners and security analysis tools
Establish security performance baselines and regression testing
Include red team exercises focused on ML system compromise
Red Flags:
ML systems deployed without any security testing or validation
Security testing that ignores ML-specific attack vectors and vulnerabilities
No integration between ML security testing and deployment gates
Penetration testing that treats ML systems like traditional applications