Try JFrog
Guides
Contact SupportTry JFrog
  1. Get Started

Secure AI Workflows: A Guide to Scaling Velocity Safely

Artificial intelligence is the new foundation of the enterprise software creation workflow. For technology leaders, the value proposition of a native DevSecOps AI strategy is clear: empower your developers to ship innovative code at unprecedented speed, while ensuring the security, provenance, and compliance of every artifact.

This guide provides a comprehensive, high-level roadmap for you to establish a native, secure DevSecOps control plane using the JFrog Platform, effectively governing both human AI consumers and autonomous agents without sacrificing speed.

The following report is structured into five core operational phases, with a summary checklist:

Phase 1: Securing the AI Supply Chain

The proliferation of open-source AI models has created blind spots for enterprise security teams. Developers frequently download unvetted models from public repositories such as Hugging Face, bypassing standard IT governance and introducing significant risks.

To counteract this, structure your organization to centralize visibility for human developers:

  • Establish a Single Source of Truth: To discover, govern, and deploy all AI models across the enterprise, use the JFrog AI Catalog. This provides a centralized hub for managing both internal models and external API providers. You can view the curation policy status on the models page: If an open-source model violates a security policy or is blocked by curation, the "Use model" and deployment action buttons are automatically disabled in the model header.
  • Configure Curation Settings for Model Packages: To safely discover and download open-source model packages (such as those from Hugging Face), enable Curation within the JFrog Administration module. This ensures that models are properly evaluated against security policies before entering the registry.

Phase 2: Governing AI Developer Environments

As developers increasingly use AI assistants and local agents that need direct access to internal enterprise systems, they rely on MCP and Agent Skills. If left unmanaged, these connections may expose the enterprise to prompt injections, where agents execute malicious instructions hidden in seemingly benign files.

To govern these agents effectively, follow these best practices:

  • Scope the Agent's Token and User: Never provide an agent with your personal admin credentials. Create a dedicated service user or a scoped access token that provides only the specific permissions needed. For instance, if the agent's role is limited to searching artifacts, it should not have the ability to deploy or delete anything.
  • Use Project-Scoped Tokens: Whenever possible, restrict the developer's token to a specific project rather than the entire platform. This significantly reduces the risk if the agent hallucinates or is compromised through indirect prompt injection.
  • Start Read-Only, Expand Gradually: When implementing a new agent, begin with read-only queries (e.g., artifact searches and build info lookups). Once you have verified the agent's reasoning and results, you can gradually add targeted write operations.
  • Review Agent Activity using Audit Logs: While agent skills are configured to ask the developer for confirmation before mutating operations, security teams should also retroactively review agent activity. Because skills set a custom user-agent header, organizations can easily filter and identify agent-driven operations within their JFrog audit logs.
  • Enforce Data Minimization and Anonymization: Transmit only what's essential during AI interactions. Implement privacy-preserving techniques that mask or pseudonymize proprietary identifiers, ensuring sensitive personal information is removed from the data pipeline before reaching external AI models.
  • Deploy Multi-Layered Content Filtering (Pre-LLM Guardrails): Establish defense-in-depth by deploying safety classifiers that inspect user inputs and agent contexts. This pre-LLM input filtering blocks prompt injections and detects Personally Identifiable Information (PII) before data leaves the environment, while output filtering blocks harmful or hallucinatory responses before they are delivered to the developer or executed by tools.
  • Govern Approved MCP Servers: Unvetted MCP servers running on developer machines represent a critical supply chain risk, as they can access local files, credentials, and internal networks. Establish a strict policy dictating which servers are approved, blocking everything else by default. The JFrog MCP Registry enforces a zero-trust default, allows for tool-level allow/deny policies, and conducts automated security scanning.
  • Treat Skills as Artifacts: When developers install agent skills from various public sources, organizations lose visibility and control. Treat skills as formal software artifacts by hosting them in a central, governed location. The JFrog Skills Repository allows effective management, version control, and access regulation for skills, using the same workflows as traditional code packages.
  • Keep Skills Updated: The AI landscape shifts rapidly. Ensure that centrally installed skills are updated regularly so that developer agents access new capabilities, receive security improvements, and maintain compatibility with expanded MCP routing protocols.

Phase 3: Governing Coding Assistants

Coding assistants drastically increase human developer productivity but introduce risks regarding architectural integrity, security blind spots, and intellectual property contamination.

To optimize the use of these tools, implement the following governance strategies:

  • Integrate Security into the IDE: Use integrations like the JFrog plugins to bring DevSecOps directly into the coding assistant's workflow. This enables developers to scan for vulnerabilities and audit dependencies before committing code.
  • Mandate Architecture-Led Reviews: AI-generated code must be explicitly tagged and rigorously reviewed by senior engineers to prevent the compounding of structural technical debt. AI assistants lack the contextual understanding needed for legacy "brownfield" environments.
  • Protect Intellectual Property: Configure assistant enterprise settings to disable telemetry data harvesting, prevent the use of your proprietary codebase for vendor model training, and enable strict duplication detection filters.

Phase 4: Securing Autonomous Agents

As autonomous agents transition into active, independent participants within your software factory, applying human-centric security rules (such as mandatory human approvals for every single action) can severely limit their utility.

To securely integrate agents into CI/CD pipelines without compromising speed, consider these guidelines:

  • Implement "Safe Outputs" and Execution Guardrails: Instead of relying on a human to approve every automated task, leverage GitHub Agentic Workflows to secure autonomous agents through a "safe outputs" pipeline. This approach programmatically restricts the agent’s write operations, such as limiting its pull requests per run and sanitizing all outputs to eliminate potentially malicious URLs or untrusted payload data before execution.
  • Deploy Composite Identities: When an agent commits code or triggers a build, traditional generic service accounts obscure the audit trail. GitLab Duo and Microsoft Azure resolve this by utilizing "composite identities." This method mathematically links the autonomous agent's service account directly back to the human developer who initiated the workflow, ensuring perfect traceability without slowing down the agent.
  • Enforce Deterministic External Controls: Never rely on an agent's system prompt (e.g., "Do not delete the database") for security. Agent security must start with a deterministic, external security configuration. This involves using strict, infrastructure-level IAM policies and gateway controls to govern exactly which APIs and tools the agent can access, completely independent of the LLM's internal reasoning.
  • Conduct Continuous Adversarial Testing (Red Teaming): Because AI foundation models are inherently probabilistic, traditional static testing is insufficient. Security teams must implement continuous AI red teaming and adversarial testing to simulate prompt injections, jailbreaks, and data extraction attempts. This ensures vulnerabilities are caught proactively and monitors for "behavioral drift" caused by evolving data sources or model updates over time, transforming security testing into an ongoing operational practice.

Phase 5: Managing Compliance and Standards

Implementing the JFrog Platform controls detailed above directly supports compliance with the major regulatory acts and industry frameworks governing enterprise AI adoption.

Organizations must structure their AI deployments to meet these obligations:

  • The European Union AI Act: The EU mandates a risk-based classification system for all artificial intelligence deployments. If your organization builds general-purpose models or deploys AI in high-stakes environments, you must prove that your data is governed continuously, your risks are managed, and your system operations are fully documented for both regulators and downstream users.
  • Forrester AEGIS (Agentic AI Guardrails for Information Security): By utilizing the JFrog Skills Repository and composite identities, you can enforce AEGIS's mandate for "Least Agency," ensuring agents operate with just-in-time privileges.
  • Gartner TRiSM (Trust, Risk, and Security Management): JFrog's centralized visibility through the AI Catalog and continuous artifact scanning fulfill the TRiSM requirement for continuous AI runtime inspection, model cataloging, and automated policy enforcement.
  • OWASP Top 10 for LLM Applications: Establishing a secure AI supply chain with MLBOMs and vulnerability scanning directly mitigates the OWASP threats of "Software Supply Chain Failures" and "Training Data Poisoning."
  • CISA and NIST "Secure by Design": Implementing out-of-the-box guardrails via the JFrog Platform ensures AI systems are treated as traditional software components. This satisfies both the classical Secure Software Development Framework (SSDF) for traditional software outlined in NIST SP 800-218, as well as the newly established AI-specific extensions detailed in NIST SP 800-218A.

The Enterprise AI Security Checklist

Before integrating new AI tools or autonomous agents into your environment, use this best practices checklist to ensure your DevSecOps guardrails are properly configured for both human and machine workflows.

For Human AI Consumers and Developers:

  1. Route all AI model downloads through the JFrog AI Catalog.
    Why: Centralizes visibility and blocks developers from downloading unvetted models directly from public repositories.
  2. Mandate a Machine Learning Bill of Materials (MLBOM).
    Why: Tracks the exact model lineage, training data, and dependencies to guarantee supply chain integrity and meet regulatory reporting requirements.
  3. Deploy JFrog Xray to continuously scan AI models.
    Why: Proactively detects malicious code, known vulnerabilities, and unsafe serialization formats before they reach the development environment.
  4. Treat Skills as Artifacts. Why: Storing developer skills in a central, controlled location like the JFrog Skills Repository ensures strict version control and access management, reducing potential blind spots.
  5. Govern your MCP Servers. Why: Unvetted Model Context Protocol (MCP) servers on developer machines pose significant supply chain risks. Using the JFrog MCP Registry ensures a zero-trust approach with tool-level allowlists.
  6. Scope the Agent's Token and start Read-Only. Why: Agents run by users should never hold personal admin credentials; scoped tokens mapped to specific projects minimize risk, starting with read-only queries before expanding to write operations.
  7. Review Agent Activity in Audit Logs. Why: While developer skills ask for confirmation before mutating operations, security teams must proactively review JFrog audit logs using custom user-agent headers to trace agent-driven actions.
  8. Track the "Trust Tax" for AI-generated code. Why: This prevents the compounding of structural technical debt by measuring how often AI code requires rewriting or verifying (the “trust tax”), rather than focusing on superficial metrics like acceptance rates.
  9. Enforce data minimization and anonymization. Why: This approach masks or pseudonymizes proprietary identifiers to ensure sensitive information is removed before reaching external model providers.
  10. Deploy multi-layered content filtering (Pre-LLM Guardrails). Why: Intercepts prompt injections and prevents data leakage by inspecting inputs and outputs before they interact with the model.

For Autonomous Agents and CI/CD Pipelines:

  1. Implement "Safe Outputs" for agentic workflows. Why: This limits the impact of autonomous actions programmatically (such as capping an agent at 3 pull requests per run) instead of relying on manual approval processes.
  2. Enforce Composite Identities (Stateless Delegation). Why: This maps an autonomous agent's service account directly back to the human developer who initiated the task, ensuring auditability across multiple service boundaries.
  3. Enforce Deterministic External Controls. Why: This secures agents by using strict infrastructure-level boundaries because Large Language Models (LLMs) cannot be trusted to self-regulate via system prompts.
  4. Mandate continuous adversarial testing (Red Teaming). Why: Given that AI models are probabilistic, continuous red teaming simulates jailbreaks and identifies behavioral drift to ensure ongoing alignment.
  5. Map all AI systems to EU AI Act risk tiers. Why: Ensures organizational compliance with strict regulatory obligations, particularly regarding transparency and documentation for high-risk systems.

Updated about 2 hours ago