Companies are shipping AI systems into products faster than they are testing them for abuse. The model gets most of the attention, but the risk is bigger than the model. Prompts, data, app logic, APIs, and the user-facing code are all targets, too.

A model that writes code, answers customers, or approves transactions can be tricked, steered, copied, poisoned, or made to expose things it should not. You cannot patch a large language model the way you patch a server. You have to probe it, constrain it, and watch it.

The new classes of AI abuse

The main attack classes are no longer theoretical. A jailbreak coaxes a model past its own guardrails into doing what it was told not to. Prompt injection slips hostile instructions into the content a model reads, so it follows the attacker instead of the operator. The instructions do not have to be visible: hidden text in a web page or a document the model ingests can carry its own commands.

Training data can be poisoned, including the vector store behind a retrieval system, so the model treats an attacker's payload as trusted knowledge.

Attackers can also copy a model by querying it enough to reproduce how it behaves, or push it off course with inputs built to confuse it. None of this looks like the traffic a rule-based control was built to judge. It can match a signature; it cannot weigh what a prompt is trying to do.

Test before you ship, defend while it runs

AI red teaming means testing an AI system the way an attacker would: trying to break its rules, abuse its prompts, leak its data, or force unsafe behavior.

The work is moving out of the research lab and into production, and securing a live system takes two layers: finding weaknesses before release, and catching abuse while it runs.

Adversarial testing has split off from general application security for a plain reason: injection, jailbreaks, and model extraction map to nothing that a code scanner or WAF was built to check.

Straiker won the dedicated AI Security Testing category for a continuous red-teaming platform that fires multi-turn adversarial attacks at a model to find its weak points before attackers do. It was also placed among the Best Cybersecurity Startup winners, which mostly tells you that the money is following the problem.

Securing the core is a separate problem. DeepKeep won in Artificial Intelligence Security for protecting the model itself and the data it runs on, the layers that a tool built for ordinary software would miss.

Then comes runtime. Impart Security took AI Runtime Defense for controls that sit in front of a live AI system and watch for abusive prompts, jailbreak attempts, and other hostile requests as they happen.

The browser is the other exposed surface, where the AI app meets the user, and an attacker gets one more place to tamper. Jscrambler was recognized in the same category for locking down the client side.

Not every AI security problem is solved by another platform. Jeremy London of Keeper Security was named AI Security Innovator of the Year for work on AI and threat analytics, the kind of judgment that does not ship as a feature.

Tools matter. So do the people who know how models break and how attackers think.

Building a continuous red-teaming loop

Testing it before you ship is no longer enough on its own. Models drift, prompts change, and new integrations appear. The work is a loop between offensive testing and runtime defense:

  1. Map the full attack surface. Do not test only the model. Include system prompts, the retrieval and vector stores, downstream API connectors, and the client-side interface. Attackers hit the weakest link.
  2. Automate adversarial testing in the pipeline. Run multi-turn attacks, injection, and jailbreak variants among them, whenever a prompt, model, or dependency changes, not once a year.
  3. Put an AI-aware layer in front of the model. Signatures do not capture intent. A gateway that scores incoming prompts for injection and jailbreak patterns, and checks outputs for data leakage and policy violations, does.
  4. Close the loop. Every successful attack found in testing or caught at runtime becomes a permanent test case and a blocked pattern. Offensive findings should feed straight back into defensive controls.

The rule for production AI is blunt: assume attackers will try to make the system ignore your rules. Test it before you ship. Watch it after it goes live. A model that has never been red-teamed is one whose weaknesses attackers will find first.

Part of The Stars Briefing, our editorial series on the trends behind the 2026 Cybersecurity Stars Awards, a program The Hacker News runs. This piece analyzes where the field is moving and uses the winners as examples. It is not a product review. The complete list of 2026 winners is live at awards.thehackernews.com/winners/2026.