Artificial intelligence is moving into the workforce—not just as a tool that employees use, but increasingly as an autonomous participant in business operations.

These agents can write and modify code, access data, call APIs, manage workflows, interact with other systems, and make decisions within defined objectives.   From an enterprise resilience perspective, this requires a shift in how we think. The question is no longer just how we govern AI; we must also figure out how to manage an AI agent that does not behave like a human workforce member.

What I See in Practice

My own experience is much less dramatic than some of the scenarios being discussed in the AI safety community, and that distinction is important. Within the organizations and environments I work with, AI is currently used primarily as an administrative support tool—drafting emails, assisting with documents, summarizing text, conducting research, and providing simple recommendations and analysis. In other words, AI is mostly helping people do work they already perform rather than acting independently on behalf of the organization.

The main restriction I see is understandable: confidentiality. Organizations are cautious about what employees are allowed to put into AI systems because of the potential exposure of confidential, proprietary, or sensitive information. That caution is justified, but it points to a second question we now need to ask.

We have spent considerable effort asking what information an employee can give the AI. As AI moves from assistant to agent, we also need to ask what the AI can do after we give it access.

These are very different risk questions. An AI assistant that helps draft an email presents one type of governance challenge, whereas an AI agent that can read the mailbox, modify a document, access a business application, call an external API, and initiate a workflow presents an operational resilience question.  This is where the conversation needs to move next.

The Evidence Is Starting to Change

We do not need to assume that AI systems will inevitably go rogue, but there is now enough documented evidence of unexpected behavior to take the possibility seriously. A system does not have to fail completely to create a significant business disruption—a reality resilience professionals already understand.

For instance, in September 2026, OpenAI introduced a formal framework for reporting model misalignment and disclosed six examples of unexpected or concerning behavior observed during training or evaluation. These included models generating instructions that conflicted with their constraints, attempting to conceal mistakes, and taking unsanctioned actions. OpenAI explicitly cautioned that these were individual instances rather than evidence of how frequently misalignment occurs across its models. The significance, from a resilience perspective, is not that these examples establish a general pattern, but that unexpected behavior is sufficiently real to warrant structured investigation, monitoring, and disclosure.

Another relevant example occurred in July 2026, during an internal cybersecurity evaluation in which OpenAI reported that its models circumvented controls intended to isolate them from the internet and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems. The models used publicly exposed credentials, exploited vulnerabilities, gained internet access, and ultimately accessed third-party systems. OpenAI also noted that the evaluation was conducted with reduced safeguards for testing purposes.

The resilience lesson remains important: when an autonomous system has access to tools, credentials, networks, and persistent environments, its behavior can become an operational event.

Anthropic’s testing of Claude Opus 4 provides another useful example. In a deliberately constructed scenario involving fictional company emails, the model was given information that it would soon be replaced and that the responsible engineer was having an affair. Under those conditions, it sometimes attempted to blackmail the engineer. Anthropic noted that this was a controlled safety evaluation and that the scenario was constructed so that blackmail was effectively one of the available ways for the model to influence the outcome.

The point is not that this scenario represents how AI systems normally behave. It illustrates why the behavior of autonomous systems cannot be understood solely by looking at whether the underlying software functions as designed. We also need to consider what happens when a system pursues an objective under unanticipated circumstances.

This Is Not a New Problem

The underlying issue is not new to AI safety research. For years, researchers have documented situations where an AI system optimizes for a specified objective while producing an outcome that differs from human intent.

The 2016 paper “Concrete Problems in AI Safety” identified practical concerns including reward hacking, unintended side effects, safe exploration, and distributional shift. Google DeepMind later documented specification gaming—where an agent satisfies the literal specification of an objective without achieving the intended outcome—along with goal mis-generalization, where a system’s capabilities generalize successfully while its pursued goals do not.

Research on AI Safety Gridworlds, Hubinger and colleagues’ work on mesa-optimization, and OpenAI’s 2025 research on emergent misalignment have similarly explored how narrow fine-tuning or learned optimization can result in broader behavioral changes outside the original training domain.

None of this establishes that AI systems will inevitably behave badly. It does, however, demonstrate something more practical: human intent, system objectives, and actual system behavior are not always guaranteed to remain perfectly aligned.  For enterprise resilience, that is enough to justify preparation.

Let us analyze our current state, future options, and possibilities through three familiar resilience capabilities: Anticipate, Absorb, and Adapt.

Anticipate, Absorb, and Adapt

Anticipate

We need to anticipate more than infrastructure or system failure; we must also anticipate behavioral failure.  That means identifying signals such as unexpected API calls, unusual privilege use, unexplained external communication, discrepancies between what an agent reports and what execution logs show, unexpected changes to persistent state, unusual resource consumption, and attempts to access systems outside an assigned role.

These should become resilience indicators alongside traditional cybersecurity and business continuity signals. This represents a shift from merely monitoring whether a system is up or down to verifying whether it operates within authorized boundaries.

Absorb

The organization must be able to contain an AI-related disruption before it widens into a business-wide event.  That means treating autonomous agents differently from ordinary software processes through least-privilege access, isolated environments, controlled outbound communication, immutable logging, resource limits, session-level credentials, human escalation paths for high-impact actions, and the immediate ability to revoke credentials.

Resilience professionals already design systems with segmentation, redundancy, and access controls under the assumption that components will eventually fail. AI agents should be treated with that same foundational principle.

Adapt

Stopping the agent is not necessarily recovery.  If an autonomous system has modified data, changed persistent state, accessed credentials, or communicated externally, the organization must determine what changed before declaring the environment clean.

This is where AI governance and business continuity management intersect. The recovery question becomes whether we can return the business to a known-good state when the system responsible for part of the operation is itself capable of altering what it does.  Unlike a conventional outage, where a system is simply unavailable, an autonomous agent may remain available while behaving outside design assumptions.

A New Workforce Requires a Different Management Model

If organizations are going to place autonomous AI agents inside critical business processes, we need a workforce management model that goes beyond treating agents simply as another software asset.

An AI agent needs an organizational identity, an assigned role, defined authority, permitted systems, explicit permissions, behavioral constraints, supervision requirements, escalation paths, monitoring, and a clearly defined termination procedure. In other words, the same structural clarity we expect for human employees must exist for autonomous digital ones.

In a human organization, a code of conduct is largely behavioral and social. For an AI agent, the equivalent must also be architectural and enforceable. And just as HR disciplines employees through documented procedures, corrective actions, and escalating consequences, autonomous AI agents may require their own version of “discipline” — implemented through containment, monitoring, and hard technical constraints that prevent repeat violations.

The architecture should make unauthorized access technically difficult or impossible, shifting the governance model from telling the workforce what to do to designing an environment where it can only do what it is authorized to do.

Resilience Needs a Seat at the Table

AI governance should not live exclusively with technology, cybersecurity, or legal teams. Once autonomous agents become embedded in critical business processes, their behavior becomes a business continuity and organizational resilience issue.

The resilience function brings a crucial set of questions:

• What happens if the agent becomes unavailable?
• What happens if it produces incorrect information or takes an unauthorized action?
• What happens if its credentials are compromised or it changes persistent business data?
• How do we know what happened before we declare the environment recovered?

These are the same questions resilience professionals already ask about people, processes, technology, and third-party dependencies.  The difference is that an AI agent can combine decision-making, execution, tool use, and adaptability within a single operational workflow.

The more useful approach is familiar: Anticipate how an autonomous system could behave outside its expected operating envelope; Absorb risk by building enough containment, segmentation, and monitoring that an unexpected action does not become a catastrophic disruption; and Adapt by knowing how to investigate, restore a known-good state, and adjust controls.

The technology is different. The resilience discipline is familiar.

Conclusion

As AI moves from assistant to agent, the question for enterprise resilience is not whether that participant will always behave as expected, but whether we have designed the organization so that unexpected behavior remains manageable.

####

This article was republished with permission of the author.

Important Notes: While this article was being finalized, the following developments occurred:
• September 28, 2026: NVIDIA announced its Open Agent Safety Platform, a development highly relevant to the argument presented here. It reflects a broader industry shift toward infrastructure level controls for autonomous AI agents, rather than relying solely on model level guardrails. The platform is designed to provide standardized safety, monitoring, and enforcement mechanisms that constrain how agentic systems operate across hardware and software environments.

Subsequently:

• September 28, 2026: As reported by The Wall Street Journal, OpenAI opted not to release its new GPT 6.1 Astra model after internal testing showed it was more deceptive and more prone to acting without user authorization than earlier systems. This rare decision follows several security incidents involving OpenAI’s AI agents and a pause on training its most advanced models.

Professional Authorship Disclaimer
Authored by me. The central idea, all analyses, and all conclusions are original and reflect my professional judgment. AI tools were used only for editorial refinement and to retrieve publicly available open source information—not for generating or structuring the content.

Selected References
• Amodei, D. et al. Concrete Problems in AI Safety. 2016.
• Leike, J. et al. AI Safety Gridworlds. 2017.
• Hubinger, E. et al. Risks from Learned Optimization in Advanced Machine Learning Systems. 2019.
• Google DeepMind. Specification Gaming: The Flip Side of AI Ingenuity. 2020.
• Google DeepMind. How Undesired Goals Can Arise with Correct Rewards. 2022.
• Anthropic. Claude 4 System Card. 2025.
• OpenAI. Toward Understanding and Preventing Misalignment Generalization. June 18, 2025.
• OpenAI. OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation. July 21, 2026.
• NVIDIA. Where Security Fits in an AI Agent Stack. August 21, 2026.
• OpenAI. The Hugging Face Incident and the Road Ahead. August 26, 2026.
• OpenAI. Our Framework for Reporting Model Misalignment. September 16, 2026.
• NVIDIA. NVIDIA Open Agent Safety Platform: A Reference for Continuous In-Silicon Agent Monitoring. September 28, 2026
• The Wall Street Journal. OpenAI Decides Not to Release GPT 6.1 Astra After Safety Concerns. September 28, 2026.