Security teams are still catching up with AI deployment. The threat models that work for traditional software don’t fully address AI systems’ unique vulnerabilities.

I’ve been tracking AI security incidents and emerging attack patterns. Some threats are now well-understood. Others remain underappreciated.

Prompt Injection: Now a Known Problem

Prompt injection - manipulating AI systems through crafted inputs - is now widely recognized. Security teams understand the risk, though defenses remain imperfect.

The pattern: an attacker inserts instructions into content the AI processes. The AI follows the attacker’s instructions rather than (or in addition to) the intended instructions.

This shows up in:

• Email assistants that read malicious emails
• Document processors that encounter crafted documents
• Chat systems that process user inputs designed to manipulate behavior

Defenses are improving. Input filtering, output validation, and architectural patterns that separate instructions from data help. But prompt injection isn’t solved - it’s an ongoing cat-and-mouse game.

Data Poisoning: The Slow Burn

Less visible but potentially more damaging: attacks on training data.

If an attacker can influence training data, they can influence model behavior. The effects may not appear until long after the poisoning occurred, making detection difficult.

Public training data. Models trained on internet data can be influenced by content strategically placed on the web. A determined actor can shift model behavior through coordinated content creation.

Crowdsourced data. Systems that learn from user feedback can be manipulated through coordinated false feedback.

Enterprise data. Internal training data may be compromised if internal systems are compromised.

Organizations training or fine-tuning models need data provenance controls and anomaly detection on training inputs. Most are unprepared for this.

Model Theft and Extraction

AI models represent significant investment. Protecting them matters:

Direct theft. If model weights are accessible, they can be stolen. This is straightforward security - access controls, encryption, monitoring.

Model extraction attacks. With enough queries to a model, attackers can extract a functional copy. The attack model doesn’t need the original weights; it just needs to reproduce behavior.

API rate limits and query pattern monitoring help. But for valuable models, assume attackers will try to copy them.

Insider threats. Employees with model access can exfiltrate weights. This isn’t unique to AI but is heightened given model value.

Inference Attacks on Private Data

AI systems can leak information about their training data:

Membership inference. Determining whether specific data was in the training set. For sensitive data, this can be a privacy breach.

Data extraction. Extracting actual training examples through careful querying. Models sometimes memorize and can regurgitate sensitive training data.

Attribute inference. Inferring information about individuals in training data that wasn’t explicitly present.

These attacks are relevant for any model trained on sensitive data - which includes most enterprise AI.

Agent-Specific Threats

AI agents that take actions introduce new attack surfaces:

Action manipulation. Tricking agents into taking harmful actions - making unauthorized purchases, sending malicious emails, modifying systems.

Privilege escalation. Agents may have access to systems that attackers can exploit through the agent. The agent becomes an attack path.

Cascading failures. In multi-agent systems, compromising one agent may enable compromising others.

Denial of service. Making agents consume resources through crafted inputs that cause expensive processing.

Organizations deploying AI agents need threat modeling specific to agent capabilities. The traditional application security model doesn’t cover agents adequately.

Supply Chain Risks

AI systems have extended supply chains with security implications:

Model supply chain. Pre-trained models from external sources may have unknown properties or embedded vulnerabilities.

Library vulnerabilities. AI frameworks and libraries have vulnerabilities like any software. Dependency management matters.

Data supply chain. External data sources used for training may be compromised.

API dependencies. AI systems depending on external APIs inherit those APIs’ security properties.

Organizational Preparedness

From what I observe, organizational preparedness for AI security varies widely:

Security teams learning AI. Most security teams are still developing AI expertise. The gap between AI deployment speed and security capability is concerning.

AI teams learning security. ML engineers often have limited security background. Security isn’t built in from the start.

Incident response gaps. Would your organization detect an AI-specific attack? Have you rehearsed AI security incidents?

Policy gaps. Security policies often don’t address AI-specific risks. What are the rules for model access, training data, external model use?

Recommendations

For organizations deploying AI:

Threat model AI specifically. Don’t assume traditional security covers AI. Map AI-specific threats for your specific systems.

Build AI security expertise. Either develop internal capability or partner with specialists. AI consultants Sydney with security experience can help assess and address AI-specific risks.

Control training data. Know where training data comes from. Monitor for anomalies. Implement provenance controls.

Protect models. Treat models as valuable assets. Access controls, monitoring, and response plans for model theft.

Design agents defensively. Limit agent capabilities to what’s needed. Implement approval flows for consequential actions. Monitor agent behavior.

Prepare for incidents. Develop AI-specific incident response plans. Practice them.

The Trajectory

AI security threats will increase as AI deployment increases. Attackers are already targeting AI systems; this will accelerate.

Organizations that build AI security capability now will be better positioned. Those that assume traditional security is sufficient will have unpleasant surprises.

The good news: awareness is growing. Security frameworks for AI are emerging. The tools and techniques for AI security are improving.

The bad news: deployment is outpacing security. Most organizations have deployed AI systems with security considerations that are inadequate for the threats.

Close that gap. Work with teams like Team400 who understand both AI and security. Because the threat landscape for AI is evolving faster than most organizations realize.