I spent a chunk of last year building anomaly detection into our SOC pipeline at a mid-sized hosting provider, and the thing that struck me wasn't how good our model got. It was how fast the attack traffic adapted to it. We'd tune a classifier to catch a pattern of credential-stuffing attempts, ship it, and within two weeks the failed-login timing distribution had shifted just enough to slip under our threshold. That's not coincidence. That's an adversary running their own optimization loop against our detector, whether or not they're using a fancy model to do it. This is what "AI vs AI" actually looks like in production, and it's worth breaking down carefully because a lot of the discourse around it is either hype or dismissal, and neither helps you build a better runbook.
What it is
AI vs AI security, stripped of marketing language, is the use of machine learning models on both sides of an intrusion attempt: attackers using generative and predictive models to scale reconnaissance, craft social engineering content, and evade detection, while defenders use models to detect anomalies, correlate signals across huge volumes of logs, and automate response faster than a human analyst could. It's not a single product category. It's a shift in how the entire attack and defense lifecycle is instrumented.
On the attacker side, this shows up as LLM-generated phishing emails that read like they were written by a native speaker who actually knows your org chart, malware that uses generative models to rewrite its own signature on each compilation pass, and reconnaissance bots that use language models to parse job postings, LinkedIn-style profiles, and public repos to build a target profile automatically. On the defender side, it shows up as behavioral baselining (what does "normal" look like for this user, this host, this service account), unsupervised anomaly detection across NetFlow and auth logs, and increasingly, LLM-assisted triage where a model reads an alert queue and drafts an incident summary before a human ever looks at it.
The important nuance: most of what gets called "AI attack" today isn't a rogue autonomous agent breaking into your infrastructure. It's automation-plus-generation applied to existing attack techniques. The techniques (phishing, credential stuffing, lateral movement, C2 beaconing) are old. The scale and personalization are new.
How it works
Let's get concrete, starting with the attacker side. A generative model doesn't need to be sophisticated to be dangerous at scale. Attackers use it for three things mainly: content generation (phishing text, fake support chat scripts, deepfake voice for vishing), evasion (polymorphic malware, obfuscated payloads that get rewritten by a model until they slip past a specific AV signature), and target selection (scraping and summarizing OSINT to prioritize which employees to phish first, based on role and likely access level).
I've seen this play out directly. We had an incident where an attacker sent a spear-phishing email to someone on our billing team, referencing an actual vendor name and a plausible invoice amount, written in a tone that matched how our actual vendor communicates. It wasn't caught by any signature-based filter because there was no signature — it was original text generated for that one target. The only thing that caught it was a behavioral rule: the reply-to domain didn't match the sending domain's usual pattern, and that mismatch triggered a quarantine.
# Simplified detection rule we used after that incident
rule suspicious_reply_to_mismatch:
condition:
from_domain != reply_to_domain
AND from_domain not in known_forwarding_services
AND recipient_department in ["finance", "billing", "payroll"]
action: quarantine_and_alert
severity: high
On the defense side, the workhorse pattern is anomaly detection over baselined behavior. You're not looking for known-bad signatures anymore, because known-bad signatures don't survive contact with a model that can regenerate the payload. You're looking for statistical deviation from what's normal for a specific entity — a user, a host, a service account, an API key.
A typical pipeline looks like this: collect auth logs, process logs, and network flow data into a time series per entity, compute a rolling baseline (mean and variance of login times, process spawn rates, outbound connection counts), and score incoming events against that baseline using something like an isolation forest or a simpler z-score threshold if you don't need the complexity. The output is a confidence score, not a binary verdict, and that distinction matters a lot operationally.
# Example anomaly scoring output from a host-behavior model
host: sw-infrarunbook-01
entity: svc-backup-agent
baseline_outbound_connections_per_hour: 4.2 (stddev 1.1)
observed_outbound_connections_last_hour: 38
anomaly_score: 0.94
top_contributing_features:
- outbound_connection_count (weight 0.61)
- new_destination_asn (weight 0.22)
- off_hours_activity (weight 0.17)
recommended_action: isolate_and_review
What makes this genuinely "AI vs AI" rather than just "AI defense against normal attacks" is when the attacker is aware of the detection model and actively probes it. We've seen credential-stuffing campaigns that clearly ran a low-and-slow variant against us first, watched what got blocked, then adjusted timing and source IP rotation to stay just under our alerting threshold. That's adversarial probing, and it's the same concept as adversarial examples in image classification, just applied to network behavior instead of pixels.
Defenders respond to that by not relying on a single static threshold. We moved from a fixed rate limit to a model that recalculates the acceptable failed-login rate per IP range dynamically, using a shorter window during periods of elevated global threat intel and factoring in ASN reputation. The point isn't that our model is smarter than theirs. It's that the cost of continuously adapting on both sides raises the bar for the attacker, and a lot of them will move to a softer target rather than keep paying that cost.
Why it matters
The economics have changed. Before generative AI, sophisticated personalized phishing required a human to spend real time researching a target. That put a natural ceiling on how many high-quality attacks a threat actor could run per day. That ceiling is mostly gone now. A single operator with API access to a capable model can generate hundreds of personalized lures in an afternoon, each referencing plausible internal details scraped from public sources.
This changes what "defense in depth" needs to mean for infrastructure teams. It's no longer enough to rely on spam filters tuned for generic phishing markers like poor grammar or mismatched branding, because those markers are disappearing. You need behavioral and contextual signals: is this the first time this sender has emailed this recipient, does the request pattern match a known business process, is there urgency language paired with a request for a deviation from normal procedure (a new bank account, a rushed wire transfer, an out-of-band password reset).
For defenders, the upside is real too. Log volume at any organization running more than a handful of services has long since outstripped what a human SOC team can manually review. I've watched a three-person SOC team go from missing lateral movement indicators buried in tens of thousands of daily auth events to catching them within minutes, purely because a model was doing the first-pass triage and surfacing the seven events out of forty thousand that actually mattered. That's not exaggeration. That's the actual ratio in a mid-sized environment.
The risk on the defense side is over-trusting the model's output without understanding its blind spots. A model trained on your last six months of "normal" traffic will happily treat a slow, patient attacker who mimics normal patterns as unremarkable, because by definition they're inside the baseline. AI defense raises the floor against noisy, high-volume attacks and does comparatively little against a patient, well-resourced adversary who studies your baseline first. That asymmetry is the single most important thing to internalize if you're building a security program around ML-based detection.
Real-world examples
A few patterns have shown up repeatedly enough across incidents I've either handled directly or reviewed post-mortems for that they're worth naming specifically.
Business email compromise using generated content. Attackers scrape a company's public filings, press releases, and LinkedIn-style employee listings, then use a language model to draft a CEO-to-CFO wire transfer request that matches the actual writing style and org structure of the target company. The tell isn't grammar anymore, it's process deviation: does this request skip an approval step that's normally required.
Polymorphic malware droppers. Rather than a fixed binary signature, some malware families now use a generation step at build time (sometimes an actual small model, sometimes just templated obfuscation) so each deployed copy has a unique hash and slightly different code structure, defeating hash-based blocklists. Defense against this has shifted almost entirely to behavioral EDR — what does the process actually do at runtime, not what does its binary look like at rest.
Automated vulnerability scanning at scale. Attackers run AI-assisted fuzzing and scanning tools that can prioritize which of thousands of discovered endpoints are most likely to be exploitable based on response patterns, cutting the manual triage time from days to hours. On the defense side, the equivalent tooling does the same thing for your own attack surface — continuous automated pentesting that flags your own exposed services before someone else's scanner does.
Deepfake-assisted vishing. Voice cloning used in a callback scam against a helpdesk, where the attacker calls pretending to be an executive requesting an emergency password reset, using a cloned voice sample pulled from a public earnings call or conference talk. We added a policy after seeing chatter about this in industry threat-sharing groups: no voice-only authorization for credential resets, full stop, regardless of how convincing the caller sounds. A callback to a verified number and a secondary factor that doesn't rely on voice recognition closes this gap almost entirely.
On the defense-innovation side, one pattern I'd point to as genuinely effective: using an LLM to write a plain-language summary of a correlated alert chain before it hits a human analyst. Instead of a queue of forty raw alerts from a suspicious lateral movement event, the analyst gets a two-paragraph narrative: what host was first affected, what credentials were used, what the likely blast radius is. That doesn't replace analyst judgment, but it cuts triage time dramatically, and triage time is usually the bottleneck in incident response, not analyst skill.
Common misconceptions
The biggest one I run into, including from people who should know better, is the idea that AI attacks mean fully autonomous AI agents independently discovering zero-days and executing multi-stage intrusions with no human involved. That's not what's happening in the overwhelming majority of real incidents. What's happening is AI-assisted automation of existing techniques, at higher volume and with better personalization. The human is still very much in the loop on the attacker's side, directing the model and picking targets. Treating this as a sci-fi scenario distracts from the boring, fixable reality: your phishing training needs to account for content that no longer has grammatical tells, and your detection needs to move from signature-based to behavior-based.
The second misconception is that deploying an ML-based detection tool is a one-time investment. It isn't. A model trained on last year's traffic patterns degrades as your infrastructure changes and as attacker behavior shifts specifically to evade it. If you're not retraining and revalidating your detection models on a regular cadence, and specifically testing them against known evasion techniques, you're running a defense that was accurate six months ago and is quietly less accurate today. We run a quarterly red-team exercise specifically targeting our own anomaly detection thresholds, not just our network perimeter, because the detection logic itself is now an attack surface.
The third misconception, and this one costs people real money, is assuming AI defense means you can reduce headcount on your SOC team. What it actually does is change what your analysts spend time on. Instead of manually correlating log entries, they're validating model output, tuning false-positive rates, and handling the genuinely ambiguous cases the model flags as uncertain. Teams that cut analyst headcount right after deploying an ML detection layer tend to end up with a slow but steady increase in missed incidents, because there was no one left to catch the cases where the model's confidence score sits in that uncomfortable 0.4 to 0.6 range that needs a human judgment call.
A fourth one worth flagging: assuming that because an attack used AI-generated content, it must be more sophisticated or harder to stop than a "traditional" attack. Sometimes the opposite is true. Generated phishing content, while grammatically clean, often has a distinct "averaged" quality to it, generic phrasing that lacks the specific idiosyncrasies of how a real person in that role would actually write. Training your team to notice that flatness, alongside the process-deviation checks I mentioned earlier, is often more effective than trying to build a detector that flags "AI-written" text directly, which is a notoriously unreliable classification problem in its own right.
Where this leaves infrastructure teams practically: don't chase the idea of a single AI system that solves security. Build layered behavioral detection, keep your baselines fresh, keep humans in the loop on ambiguous cases, and assume that whatever detection logic you ship today will get probed and adapted against within weeks by anyone motivated enough to care. The race doesn't have a finish line. It has a cadence, and staying in it is the actual job.