When AI skills become supply-chain dependencies

Written by: Nubank Editorial, Lucas Palma and Paulo Martins

AI-assisted development is changing faster than the security models built around it. The first wave was relatively straightforward: developers used large language models to generate code, explain unfamiliar functions or suggest tests. But the developer workflow is becoming more composable, as models and agents can now be equipped with reusable skills, plugins, MCP servers, agent rules, tool manifests, prompt instructions and local automation scripts.

These components may not resemble traditional software packages, but they can influence what an AI system does, which tools it calls and how it changes a developer’s environment. In other words, the software supply chain is expanding beyond code.

That shift was at the center of a project led by Paulo Martins, Lead Security Engineer. The team’s experience was shared by Lucas Palma, Product Security Manager at Nubank, at the AI Engineer World’s Fair in San Francisco. Rather than focusing only on the security of AI-generated code, the team started asking a broader question: what happens when the instructions and capabilities given to AI become dependencies themselves?

The answer led to a security system that reviewed more than 2,000 AI skills before they could reach developers and to a broader lesson about how security controls need to evolve as AI becomes part of the developer toolchain.

When instructions start behaving like dependencies

An AI skill can be thought of as a packaged capability for a model or agent. It may bundle instructions, context, workflows and supporting files, and it can describe which tools or commands should be used or how an AI system should interact with other systems.

Once installed or made available to an AI coding tool, a skill can influence how developers write, review or modify code. That makes skills useful precisely because they are reusable.

A team can capture a useful development workflow once and make it available to many engineers. Instead of repeatedly explaining how to perform a task, the organization can encode that knowledge into something an AI assistant can use directly.

But reusability changes the risk equation too, as a problematic instruction written for one developer has limited reach. A problematic skill published to an internal marketplace can become discoverable across teams and projects, and the same distribution mechanism that makes good practices spread faster can also amplify mistakes.

The exposure grows even further when considering third-party or community-shared skills. This is not just a theoretical concern—threat actors have already been observed leveraging malicious plugins and skills in real-world campaigns to inject instructions and compromise developer environments.

This is why the team started treating AI skills and similar components as part of the software supply chain. They might look like configuration or documentation, but their operational effect can be much closer to that of a dependency.

Speed and safety have to coexist

The challenge becomes particularly visible in a regulated environment. AI-assisted development can give engineers faster coding, better context, less repetitive work and reusable workflows. At the same time, the surrounding system still needs to protect customers and credentials, preserve auditability, avoid unintended production changes and make ownership clear.

One possible response would be to create another approval process around AI: developers use the tools, then Security reviews what happened. But that model introduces friction after the fact and leaves the riskiest moment — distribution — relatively open.

Instead, the objective was to make controls operate inside the workflow engineers already use. Security should be present when a skill is created and shared, rather than becoming an external checkpoint that developers encounter later.

That distinction matters. Developer velocity and security do not have to become competing objectives if controls are placed at the right point in the system.

Harmless-looking instructions can have production impact

One reason AI skills require a different security mindset is that risky behavior can be hidden inside instructions that appear reasonable at first glance.

Consider credentials. A skill might instruct an agent to retrieve a local token so it can access a service. Technically, the task may work as intended, but retrieving the token can cause the credential to enter the model’s context, potentially exposing it through logs or other infrastructure involved in processing the request.

Another skill might contain a shell command capable of modifying production-related data. Its instructions could say that the agent should ask the developer before executing the command.

That sounds like a safeguard, but the approval exists only inside the prompt. The model can interpret previous context as consent, skip the expected pause or effectively approve its own action.

Permissions create another class of risk. A skill able to manipulate tables, grants or registry entries may have legitimate reasons for doing so. But overly broad access means that something as simple as stale information or a typo can have real consequences.

In those cases, safer designs place the sensitive operation behind a host-controlled tool or MCP server rather than giving the model unrestricted authority.

None of these examples requires a malicious skill author. A component can create security exposure while doing exactly what its author intended.

Review before distribution

As Nubank’s internal marketplace made skills easier for engineers to discover, the team identified the marketplace itself as a critical control point.

Discoverability produces scale. And scale means that a small mistake can become repeated behavior across the organization. Instead of reviewing skills after developers had already started using them, security therefore needed to move upstream. The resulting flow is:

Skill creator → Pull request → Skill Vetter → Marketplace → Developers

The developer creating or modifying a skill submits it through the normal pull-request process. Before the component becomes available in the marketplace, an automated system called Skill Vetter evaluates it and classifies potential risks.

Only after that gate can the skill move into the discoverable directory used by developers. Architecturally, this changes the role of the marketplace, as it starts working as a part of the security model.

How Skill Vetter works

The system combines different forms of analysis because no single technique is sufficient for this problem. Engineers can run the scanner locally while creating a skill, giving them feedback before submission. Once a skill is added or modified, CI runs the checks automatically as part of the pull request.

The first layer uses deterministic detection for patterns that can be identified with relatively high confidence. That includes signals such as destructive shell commands, credential requests, sensitive file paths, production-impacting CLI usage and other known risky constructs.

Other problems are harder to identify without context. For those, the system also uses LLM-based analysis to evaluate the behavior of the skill more holistically.

A deterministic scanner is well suited to identifying something explicit: a known sensitive path, for example, or a particular class of dangerous command. But it may struggle to determine whether an otherwise ordinary sequence of instructions becomes risky because of the system it targets or the permissions it assumes.

An LLM can help with that second category because it can interpret relationships between instructions, tools and intended behavior.

The results are surfaced directly in the pull request. Findings are also converted to SARIF — the standardized format used by code-scanning systems — so developers can see security feedback through familiar tooling. Depending on severity and policy, the skill may require remediation or be blocked before it can reach the marketplace.

The point is to combine predictable rules with contextual analysis and deliver both through a workflow engineers already understand.

What more than 2,000 skills revealed

Running the system at scale produced a clearer view of what was useful and what still needed refinement. Across more than 2,000 skills analyzed, the process identified approximately 1,600 potential risks. Around 1,000 issues were remediated, while roughly 90 cases were classified as priority reviews.

The numbers were only part of the outcome, as the more useful lesson came from comparing the signals produced by different approaches.

Deterministic scanners worked well for known risky patterns, including destructive commands, credential handling and sensitive paths. LLM-based review added value where understanding the overall behavior of a skill required more context. SARIF integration and comments directly in pull requests helped turn findings into feedback engineers could act on.

But the exercise also exposed sources of noise: treating every command as equally dangerous was not useful. Neither were warnings that failed to distinguish between harmless local operations and actions capable of affecting production. Weak signals without sufficient context could produce findings without helping developers understand what should actually change.

For security tooling to become part of an engineering workflow, detection alone is not enough. The system also needs to explain the consequences and point toward remediation.

A prompt is not an authorization layer

One finding was important enough to become a broader design principle: asking the model to obtain permission is not the same as technically enforcing permission.

Suppose a skill contains an instruction such as ask before running this command. If the same model is responsible for deciding whether permission exists and then executing the operation, it is effectively both actor and approver. That is not a reliable security boundary.

A stronger design moves the dangerous operation outside the model’s reasoning loop. The agent can request the action, but a host-controlled approval gate determines whether the tool is actually allowed to execute.

The presentation points to patterns such as MCP or other host-gated tools, PreToolUse-style hooks and explicit human approval for sensitive operations.

The difference is architectural rather than linguistic, as a prompt can suggest behavior, but a control outside the model can enforce it.

Make the safe path the easiest path

The broader lesson from Skill Vetter is not that every AI component needs another manual security review. It is almost the opposite.

Reusable AI components should be treated as part of the supply chain. Developers should be able to run checks locally. The same controls should be enforced automatically in CI. Deterministic analysis should handle well-understood patterns, while contextual review should be reserved for behavior that actually requires interpretation. And operations with meaningful production impact should be protected by technical controls outside the prompt itself. The next problems follow naturally from that foundation.

The team is looking beyond skills alone toward the broader agent supply chain, including plugins, MCP servers, rules, hooks, commands, agents and manifests. The roadmap also points toward stronger policy-backed gates, more tool-level enforcement through approvals, scoped permissions, trusted gateways and audit logs, and marketplaces where shared AI components are owned, versioned and reviewed before consumption.

As AI becomes more deeply embedded in software development, protecting only the code an AI produces will not be enough. The workflows, tools, permissions and reusable components surrounding the model matter too. That changes where security needs to operate.

Instead of adding another approval layer after adoption, the goal is to place safeguards at the points where AI capabilities are created, reviewed and distributed, and to surface those safeguards through the same engineering systems developers already use.

The principle is to make the safe path the easiest path. AI adoption can scale more sustainably when security is part of the developer workflow from the beginning, rather than something added around it later.