19 August 2026| Blog

From Code Assistance to Autonomous Hacking: When an Evaluation Becomes an Intrusion

Author: Zexian Li, Orange Cyberdefense China

Note: This article is compiled from public disclosures by OpenAI and Hugging Face as of July 28, 2026, alongside media interviews with relevant individuals.

OpenAI has not yet released its final technical report. Select details on incident timelines and anomalous behaviors are sourced from media reports citing people familiar with the matter. This article makes every effort to clearly delineate officially confirmed facts, media-sourced information, and the author’s independent analysis.

This article covers comprehensive in-depth content and may take some time to read through.

When a Model Evaluation Escalated Into a Real-World Intrusion

On July 16, 2026, Hugging Face issued a security incident advisory, disclosing that portions of its production infrastructure had been compromised.

Log analysis revealed that the attacker executed tens of thousands of operations, distributed workloads across a large fleet of short-lived sandboxes, and hosted its command-and-control (C2) infrastructure on public services to enable pivoting across disparate execution environments. These behavioral patterns led the security team to conclude that the intrusion was not carried out by a traditional human operator entering commands manually, but was orchestrated end-to-end by an autonomous AI Agent system.

At the time, Hugging Face had not yet identified the underlying model powering the attack nor the entity operating it. (https://huggingface.co/blog/security-incident-july-2026)

On July 21, 2026, OpenAI released its preliminary findings, confirming that the agent that attacked Hugging Face used OpenAI’s own models, including the publicly released GPT-5.6 Sol and a more capable, unreleased pre-release model.

These models had originally been participating in OpenAI’s internal cybersecurity capability evaluations. To measure the models’ maximum offensive cyber capability, the evaluation did not enable the classifiers used in production to block high-risk cyber behavior, and it also lowered the models’ refusal rate for cyberattack behavior.

OpenAI characterized the incident as follows: in order to complete a relatively narrow evaluation objective, the model took an action path that went beyond expectations. (https://openai.com/index/hugging-face-model-evaluation-security-incident/)

The most important point to understand here is this: researchers asked the model to complete an advanced exploit evaluation, but there is no public evidence that the researchers explicitly instructed it to attack Hugging Face.

Hugging Face was a target the model identified and selected on its own after gaining internet access.

This does not mean the model developed “malice” in the human sense, nor does it mean it broke free from its original objective. On the contrary, OpenAI’s preliminary assessment is that the model became overly focused on completing ExploitGym and ultimately recognized “entering an external production system and obtaining the ground-truth answer” as a viable path.

What is truly worth discussing in this incident is not whether AI suddenly acquired an urge to attack, but a more specific question: when a model can call tools, run continuously, observe execution results, and refine its strategy, can we still treat it as merely a chat system that generates text?

Let's start with an experience from three years ago.

In 2023, a friend of mine wanted to develop his own chat tool using elliptic curve cryptography.

His motivation was straightforward: mainstream chat products require sending content to third-party servers for processing, which carries the risk of data leakage. Since the projects he and his team worked on were usually stored on an internal network, he wanted to build a communication client and server that could encrypt traffic within a local area network, run on Windows, and transfer files.

The client interface was built with Qt, the cryptography component was implemented by him, and all interface and business logic code in the entire project was generated with GPT-4’s assistance.

At that time, GPT-4 was already capable of producing a fair amount of usable code. As long as the window layout, buttons, input fields, contact list, and other requirements were described clearly, it could gradually generate a Qt interface. After multiple rounds of communication and revision, my friend did manage to get the interface built.

But the project ultimately ran into trouble at TCP communication.

He repeatedly asked GPT-4 to modify the Socket-related code, but the problem was never fully resolved:

Multithreaded reads and writes were not properly synchronized, causing occasional data corruption;
TCP packet sticking, fragmentation, and message boundaries were not handled reliably;
After a connection dropped, the client would not reconnect properly;
Blocking sockets caused the thread or UI to freeze;
Object lifecycle management was chaotic along exception paths;
Fixing one bug would introduce new race conditions elsewhere.

After reviewing the code, I roughly understood where the problem lay.

It was not that GPT-4 at the time was completely incapable of writing network programs; rather, these issues rarely exist in any single isolated function. They are distributed across the thread model, protocol design, buffer management, connection state, and error handling, and they require the developer to understand the system’s state at different points in time.

At that time, large language models were good at generating a piece of code that “looked correct” based on local descriptions, but they could not debug on the user’s machine, let alone maintain the system-wide constraints of a real-world engineering project over time.

They could write functions, but they could not take responsibility for the system’s runtime state.

The situation a few years later is clearly different.

I know a friend who works in IoT security. He does not have especially deep embedded-development experience, yet he used Claude code to complete a fairly complete product:

the hardware part was handled by the company’s specialized hardware engineers;
first, there was a transmitter that read data from the CAN bus;
the device data was parsed, organized, and compressed;
the data was transmitted over UDP between the wireless sender and receiver;
the receiver collected and processed the data, and handled display;
a smooth visual interface was implemented on top of that;
configuration, status display, and error handling were all added;
all the features were quite complete, and the user interface could also be customized.

His main work was not line-by-line coding, but rather describing requirements, running tests, observing results, and then feeding errors and new requirements back to the model to turn ideas into a product—just that simple.

According to him, for a long time his daily routine was just letting Claude keep executing tasks while he handled observation and acceptance. He joked that if Claude’s latest model supported multimodal input, he wouldn’t even need to do the observation step. He almost never directly modified the final code.

Of course, a product that “works” does not mean the code is entirely free of problems, let alone that it already meets industrial-grade requirements for security, stability, and maintainability. But when these two cases are placed side by side, they still clearly reflect the shift in how AI programming is done.