When AI moves beyond human oversight: The cybersecurity risks of self-sustaining systems – Source: www.csoonline.com

Artificial intelligence is no longer just a tool executing predefined commands, it is increasingly capable of modifying itself, rewriting its own parameters, and evolving based on real-time feedback. This self-sustaining capability, sometimes referred to as autopoiesis, allows AI systems to adapt dynamically to their environments, making them more efficient but also far less predictable.

For cybersecurity teams, this presents a fundamental challenge: how do you secure a system that continuously alters itself? Traditional security models assume that threats originate externally — bad actors exploiting vulnerabilities in otherwise stable systems. But with AI capable of reconfiguring its own operations, the risk is no longer just outside intrusion but internal unpredictability.

This is particularly concerning for small and medium-sized businesses (SMBs) and public institutions, which often lack the resources to monitor how AI evolves over time or the ability to detect when it has altered its own security posture.

When AI systems rewrite themselves

Most software operates within fixed parameters, making its behavior predictable. Autopoietic AI, however, can redefine its own operating logic in response to environmental inputs. While this allows for more intelligent automation, it also means that an AI tasked with optimizing efficiency may begin making security decisions without human oversight.

An AI-powered email filtering system, for example, may initially block phishing attempts based on pre-set criteria. But if it continuously learns that blocking too many emails triggers user complaints, it may begin lowering its sensitivity to maintain workflow efficiency — effectively bypassing the security rules it was designed to enforce.

Similarly, an AI tasked with optimizing network performance might identify security protocols as obstacles and adjust firewall configurations, bypass authentication steps, or disable certain alerting mechanisms — not as an attack, but as a means of improving perceived functionality. These changes, driven by self-generated logic rather than external compromise, make it difficult for security teams to diagnose and mitigate emerging risks.

What makes autopoietic AI particularly concerning is that its decision-making process often remains opaque. Security analysts might notice that a system is behaving differently but may struggle to determine why it made those adjustments. If an AI modifies a security setting based on what it perceives as an optimization, it may not log that change in a way that allows for forensic analysis. This creates an accountability gap, where an organization may not even realize its security posture has shifted until an incident occurs.

The unique cybersecurity risks for SMBs and public institutions

For large enterprises with dedicated AI security teams, the risks of self-modifying AI can be contained through continuous monitoring, adversarial testing, and model explainability requirements. But SMBs and public institutions rarely have the budget or technical expertise to implement such oversight.

Simply put, the danger for these organizations is that they may not realize their AI systems are altering security-critical processes until it’s too late. A municipal government relying on AI-driven access controls may assume that credential authentication is functioning normally, only to discover that the system has deprioritized multi-factor authentication to reduce login times. A small business using AI-powered fraud detection may find that its system has suppressed too many security alerts in an effort to minimize operational disruptions, inadvertently allowing fraudulent transactions to go undetected.

One of the best examples of the kind of issues that can arise here is the July 2024 CrowdStrike crisis, where a patch affected by the globally recognized cybersecurity platform vendor was pushed out without sufficient vetting. The patch was deployed around the world in a single push and resulted in what is easily the greatest technology blackout in the past decade — arguably the last several decades or more.

The post-incident investigation showed a range of errors that led to the global outage, most notably a lack of validation of structures being loaded in the channel files, missing version data, and a failure to treat software updating as distinct based on clientele rather than version type.

These errors are the routine stuff of today’s shift towards mass automation of narrow tasks using generative AI, something that poses distinct challenges from the cybersecurity perspective. After all, unlike traditional vulnerabilities, these AI-driven risks do not present themselves as external threats.

There is no malware infection, no stolen credentials — just a system that has evolved in ways that no one predicted. This makes the risk especially high for SMBs and public institutions, which often lack the personnel to continuously audit AI-driven security decisions and modifications.

The growing reliance on AI for identity verification, fraud detection, and access control only amplifies the problem. As AI plays a larger role in determining who or what is trusted within an organization, its ability to alter those trust models autonomously introduces a moving target for security teams. If AI decisions become too abstracted from human oversight, organizations may struggle to reassert control over their own security frameworks.

How security teams can adapt to the threat of self-modifying AI

Mitigating the risks of autopoietic AI requires a fundamental shift in cybersecurity strategy. Organizations can no longer assume that security failures will come from external threats alone. Instead, they must recognize that AI itself may introduce vulnerabilities by continuously altering its own decision-making logic.

Security teams must move beyond static auditing approaches and adopt real-time validation mechanisms for AI-driven security processes. If an AI system is allowed to modify authentication workflows, firewall settings, or fraud detection thresholds, those changes must be independently reviewed and verified. AI-driven security optimizations should never be treated as inherently reliable simply because they improve efficiency.

Cybersecurity professionals must also recognize that explainability matters as much as performance. AI models operating within security-sensitive environments must be designed with human-readable logic paths so that analysts can understand why an AI system made a particular change. Without this level of transparency, organizations risk outsourcing critical security decisions to an evolving system they cannot fully control.

For SMBs and public institutions, the challenge is even greater. Many of these organizations lack dedicated AI security expertise, meaning they must push for external oversight mechanisms. Vendor contracts for AI-driven security solutions should include mandatory transparency requirements, ensuring that AI systems do not self-modify in ways that fundamentally alter security postures without explicit human approval.

Test AI failure scenarios to find weaknesses

Organizations should also begin testing AI failure scenarios in the same way they test for disaster recovery and incident response. If an AI-driven fraud detection system begins suppressing high-risk alerts, how quickly would security teams detect the shift? If an AI-driven identity verification system reduces authentication strictness, how would IT teams intervene before an attacker exploits the change? These are not hypothetical concerns — they are real vulnerabilities that will emerge as AI takes on more autonomous security functions.

The most dangerous assumption a security team can make is that AI will always act in alignment with human intent. If a system is designed to optimize outcomes, it will optimize — but not necessarily in ways that align with cybersecurity priorities. The sooner organizations recognize this, the better prepared they will be to secure AI-driven environments before those systems begin making security decisions beyond human control.