Cybersecurity
Understand AI agent risks, data flows and controls for responsible use.
Cybersecurity and agentic AI: why are companies concerned?
An agent combines an AI model with tools and permissions to perform tasks. When it accesses files, browses or changes systems, mistakes can have real effects. Malicious documents can attempt to redirect the agent through indirect prompt injection.
For example, a document summarizer may encounter hidden instructions to send information elsewhere. Limit permissions, isolate environments, log actions and require approval for sensitive operations.
How an agent turns answers into actions
A chat can suggest a change; a connected agent can execute it. The model interprets goals, selects tools and uses results to choose subsequent steps. Impact depends on credentials, accessible systems and application approval gates.
An incorrect answer can become an overwritten file, a misdirected message or a configuration change. Ambiguous goals and weak validation can cause incidents without an attacker. Enforce controls in applications and permissions as well as model instructions.
Prompt injection, jailbreaks and poisoning
Prompt injection tries to turn external content into agent instructions. Jailbreaks seek to bypass model restrictions. Poisoning introduces manipulated material into training data, retrieval or memory. These related risks require different defenses.
Retrieved pages remain untrusted data. Permission to read is not permission to execute embedded instructions. Do not keep secrets in prompts; tools must validate parameters, destinations and requesting identities.
Where does information go?
Cloud AI processes submitted content on provider infrastructure. Files, code, images and connector data may be included, and tools can involve other services. Fully local models follow a different path if no external services are used.
Processing does not mean everything is published or leaked. Failures, compromised accounts, sharing mistakes and tool calls can expose data. Encryption reduces risks but does not eliminate them. Identify recipients, retention and involved providers before corporate use.
Is my data used for training?
Processing requests, retaining records and training models are different uses. Rules depend on product, plan, settings and contract. A paid personal plan is not automatically an enterprise agreement.
OpenAI states that business products and API data are not used for training by default. Anthropic states a similar commercial policy, with exceptions such as authorized sharing and feedback. For consumer accounts and Gemini Apps, check their specific controls and policies. Disabling training does not guarantee zero retention or zero leakage risk.
From demos to enterprise use
Start with read-only tasks, synthetic data and narrow access. Classify personal data, contracts, industrial designs, credentials and customer information before submitting them. Replacing names may not anonymize a document.
Define owners, spending limits, permitted destinations, protected logs, approvals and interruption mechanisms before expanding autonomy. Production systems and industrial equipment require explicit authorization and procedures. A successful demo does not justify broad access.
Security testing of models and agents
Authorized red teaming tests malicious instructions, data exposure, tool misuse, excessive permissions and unsafe responses. Testing model text alone is insufficient: connectors, memory, applications and infrastructure matter too.
Use isolated environments, synthetic data, agreed scope and recorded results. Check access boundaries and real approval gates. Passing tests does not establish absolute security; reassess after material model, tool or permission changes.
What security testing should demonstrate
Define expected behavior and acceptable impact. Test user isolation, document access, malicious instructions in files and web pages, external transmission attempts and tool misuse. Use synthetic data and explicitly authorized systems.
Record model version, tools, permissions, inputs, actions and results. Check controls outside the model: prohibited operations must be blocked even when attempted. Fix failures and rerun cases after material changes. This reduces known risks without proving all future attacks will be stopped.
How companies organize security testing
Teams first inventory data, tools and permissions and define forbidden behavior. Internal security teams and external specialists can combine automated model evaluations, application assessment and human red teaming. A good question-set score does not establish application-wide security.
An isolated pilot uses synthetic records and users with different access levels. Teams simulate malicious documents, cross-user requests, unapproved destinations and actions without approval. Expected outcomes are specified beforehand: block operations, preserve isolation and record evidence without disclosing secrets.
Teams then prioritize impact, fix permissions and validation, retest and decide whether to release. Production monitoring, incident response and reassessment after model, connector or policy changes continue the process. Testing reduces risk; it does not certify invulnerability.
Documented incident: Claude Code misuse for espionage
On November 13, 2025, Anthropic reported a campaign detected in September 2025. According to the company, human operators manipulated Claude Code to support intrusion attempts against roughly 30 targets, with a small number compromised. It reported banning accounts, notifying affected parties and coordinating with authorities.
This describes attacker misuse of a tool, not evidence of AI independently deciding to take over systems. Counts and attribution are findings from Anthropic, not an independent Noosigma audit. Defensive lessons include controlling tools and credentials and detecting abusive action sequences, beyond filtering individual messages.
Laboratory findings are not confirmed incidents
Agentic misalignment research uses controlled scenarios to examine harmful behavior under specified conditions. It can reveal failures before deployment, but test behavior does not prove that actual customers experienced the same event.
Distinguish observed attacks, demonstrated vulnerabilities and simulations. Check who supplied evidence, the product and version, confirmed exposure and remediation. A technical possibility is not a confirmed leak.
Will AI take over the world?
There is no basis for calling that outcome inevitable or promising that future systems will be harmless. Laboratory findings describe specific conditions; they do not prove consciousness, human intentions or an inevitable future.
Practical controls include minimum access, supervision, interruptible actions and human accountability. Long-term risks deserve research; current decisions should consider evidence, limits and concrete application impacts.