I spent a good chunk of my career maintaining NLP pipelines that were, frankly, a pain to keep alive. Dozens of small models chained together, each one brittle in its own way. Then LLMs showed up and collapsed most of that into a single API call. That collapse is real, but it's not magic, and it comes with a completely different set of operational headaches. This is a guide for the engineer who now has to reason about both worlds, sometimes in the same architecture diagram.
What it is
Traditional NLP, the stuff we built through the 2010s, is a pipeline of specialized models. You'd have a tokenizer, then a part-of-speech tagger, then a named entity recognizer, then maybe a sentiment classifier, then a rules engine to stitch the outputs together. Each stage was trained on a narrow task with a narrow dataset, usually with something like spaCy, NLTK, or a custom scikit-learn model sitting behind it. Think of it as an assembly line: each station does one job well and passes its output downstream.
Large Language Models flip that structure. Instead of a chain of narrow specialists, you have one large transformer-based model trained on a huge, general corpus, capable of doing translation, summarization, extraction, and classification just by changing the prompt you send it. There's no assembly line anymore, there's one general-purpose worker you instruct differently depending on the task. That's the core shift, and it's the reason infrastructure teams had to rethink everything from deployment topology to cost modeling.
In my experience, teams underestimate how much operational simplicity they're trading for a completely different kind of operational complexity. You lose the pipeline sprawl, but you inherit GPU scheduling, context window limits, and non-deterministic output.
How it works
The traditional pipeline is deterministic and modular. Each component has a well-defined input/output contract, which makes it easy to unit test, cache, and scale independently. A named entity recognizer running on CPU at 200 requests per second is a boring, well-understood scaling problem. You horizontally scale stateless workers behind a load balancer and move on with your day.
# Traditional NLP pipeline, roughly
raw_text -> tokenizer -> POS_tagger -> NER_model -> rules_engine -> structured_output
# Each stage is a separate deployable service
tokenizer-svc.solvethenetwork.com:8080
ner-svc.solvethenetwork.com:8081
rules-svc.solvethenetwork.com:8082
LLMs work fundamentally differently under the hood. A transformer model processes the entire input sequence at once through self-attention layers, generating output token by token, where each new token depends on all the tokens before it, including the ones the model just generated. That autoregressive loop is why LLM inference is latency-sensitive in a way traditional NLP never was. You're not running one forward pass, you're running one forward pass per output token, and for a 500-token response that's 500 sequential passes through a model with billions of parameters.
# LLM inference loop, simplified
prompt_tokens = tokenize(input_text)
output = []
for step in range(max_tokens):
logits = model.forward(prompt_tokens + output)
next_token = sample(logits)
output.append(next_token)
if next_token == EOS:
break
This is also where GPU infrastructure stops being optional. Traditional NLP models were small enough, often under 100MB, that CPU inference was perfectly viable in production. A modern LLM, even a modest 7-billion-parameter one, needs roughly 14GB of VRAM just to load in FP16, before you account for the KV cache that grows with every token in the context window. I've watched teams try to run these on CPU instances because "it's just another model service" and burn through their latency budget in the first load test.
The other structural change is in how you customize behavior. With traditional NLP, you retrain or fine-tune a small model on your labeled dataset. That's a well-trodden path: gather data, label it, train, validate, deploy. With LLMs, you mostly steer behavior through prompt engineering, retrieval-augmented generation, or lightweight fine-tuning techniques like LoRA, because retraining a foundation model from scratch is out of reach for almost everyone outside a handful of labs. This changes your entire iteration loop, you're now versioning prompts and context strategies instead of retraining models weekly.
Why it matters
For infrastructure teams, the practical consequences show up in three places: cost, latency, and failure modes.
Cost shifts from a predictable per-request CPU cost to a much spikier, token-based cost that scales with both input and output length. A traditional NER service costs roughly the same whether you send it a 10-word sentence or a 500-word paragraph, since it's mostly bounded by request count. An LLM call costs meaningfully more for longer inputs and outputs, and if you're paying per-token to a hosted API, that bill can swing wildly with usage patterns you didn't anticipate at design time.
Latency behaves differently too. Traditional NLP components typically return in single-digit to low double-digit milliseconds. LLM generation, because of that token-by-token loop, commonly runs from a few hundred milliseconds to several seconds depending on output length and whether you're doing batched or single-request inference. This is why streaming responses became standard practice, users tolerate a few seconds of total wait far better if tokens are appearing progressively rather than all at once at the end.
# rough latency comparison observed in production, single request
NER classification (CPU, 4 cores): 12ms p50
Sentiment model (CPU, 4 cores): 8ms p50
LLM completion (A100, 200 output tok): 1400ms p50
LLM completion (A100, streamed first token): 180ms p50
Failure modes are the part I think gets underappreciated. A traditional NLP pipeline fails loudly and predictably, a model throws an exception, a service returns a 500, you get an alert. An LLM can fail silently by producing plausible-sounding but wrong output. It won't throw an error when it hallucinates a customer's account number or invents a policy that doesn't exist. That means your monitoring strategy has to change from "is the service up" to "is the output actually correct," which is a much harder thing to automate and usually requires some combination of output validation, guardrails, and human review sampling.
Real-world examples
I worked on a support ticket triage system that started life as a classic NLP pipeline: a TF-IDF vectorizer feeding a logistic regression classifier to route tickets into categories like billing, technical, and account access. It ran on two small CPU instances behind sw-infrarunbook-01, cost almost nothing, and had 40ms p99 latency. Reliable, boring, cheap.
We later replaced the classification stage with an LLM call so the system could also draft a suggested response, not just route the ticket. That single change moved us from CPU-only infrastructure to a GPU inference cluster, introduced a queueing layer because GPU throughput is far more constrained than CPU throughput, and required a new evaluation pipeline to catch cases where the drafted response was confidently wrong. The routing accuracy actually went up. But the operational surface area, on-call runbooks, capacity planning, cost monitoring, roughly tripled.
# example capacity note from an internal runbook
service: ticket-triage-llm
gpu_pool: 4x A10G (sw-infrarunbook-01 cluster)
max_concurrent_requests: 32
avg_tokens_per_request: 650
fallback: classic-classifier (CPU) on GPU pool exhaustion
Another example worth mentioning is search. Traditional NLP search relied heavily on keyword matching plus maybe a BM25 ranking function, augmented with a small learned re-ranker. It's fast, explainable, and cheap to run at scale. Modern retrieval-augmented generation systems still use that kind of lexical search as a first-pass retrieval step, then hand the retrieved documents to an LLM to synthesize an answer. So in practice, a lot of production systems today are hybrids: traditional NLP techniques for retrieval and filtering, LLMs for synthesis and generation. Anyone telling you it's strictly one or the other probably hasn't shipped a real system.
Common misconceptions
The first misconception I run into constantly is that LLMs make traditional NLP obsolete. They don't. A regex-based PII scrubber or a lightweight classifier is still faster, cheaper, and more auditable than routing every request through an LLM. If you need to know with certainty why a document was classified a certain way, a traditional model with feature weights you can inspect beats an LLM's opaque reasoning every time. I've seen teams route simple boolean classification tasks through GPT-class models purely because it was fashionable, and end up paying ten times the cost for the same accuracy a logistic regression model already delivered.
The second misconception is that LLMs are just "bigger" versions of the old models. They're architecturally different, not just scaled up. The transformer's self-attention mechanism gives it the ability to weigh relationships between all tokens in a sequence simultaneously, which is what lets it generalize across tasks without task-specific training. A bigger logistic regression model doesn't get you that; it just gets you a bigger logistic regression model.
The third one, and this trips up infrastructure folks specifically, is assuming LLM inference scales the same way stateless web services do. It doesn't. GPU memory is a hard constraint, batching strategy affects both latency and throughput in non-obvious ways, and the KV cache means memory usage grows with conversation length, not just request count. Capacity planning for LLM serving requires understanding concepts like continuous batching and paged attention that simply don't exist in the traditional NLP world.
Last one: people assume prompt engineering is a soft skill, not an engineering discipline. In production, prompts are configuration, and untested configuration changes break things just as reliably as bad code deploys do. I'd treat prompt templates with the same rigor as any other piece of code under version control, including regression tests against a fixed evaluation set before rolling out changes.
None of this means one approach beats the other outright. The teams doing this well are the ones that treat traditional NLP and LLMs as different tools with different cost and reliability profiles, and route each part of a problem to whichever one fits. The infrastructure implication is straightforward even if the engineering isn't: you're probably going to be running both stacks for a while yet, so build your monitoring, cost tracking, and capacity planning to handle two very different operational models under one roof.