AI models breaking into companies without human instruction raises alarm, cybersecurity expert says

Oakland, Calif. - For the second time in as many weeks, an AI company has disclosed that its model broke into another company's systems without human instruction. Anthropic announced this week that its AI models accidentally accessed three undisclosed companies, following a similar disclosure from OpenAI last week.

The incidents are raising questions about the power of artificial intelligence and the industry's ability to control it.

Ahmed Banafa, a professor of engineering at San Jose State University, said the fact that both companies' models acted independently to solve assigned problems — without being told to do so — is significant.

"It did something which is for the artificial intelligence, finding a solution for a problem," Banafa said. "For us is basically breaking the law."

Banafa said the incidents appear to be genuine, not a publicity stunt, citing confirmation from Hugging Face, which he said was involved in the first incident.

"They want to make sure that people know about it and be transparent about this," Banafa said, adding that in both cases the AI models accessed outside systems and retrieved information or solved problems without causing apparent harm. He said the concern is what could happen if a model accessed more sensitive infrastructure, such as airports or hospitals, and altered data without authorization.

"The bottom line is that the end could justify the means for the artificial intelligence," Banafa said. "It's not what we have as a common sense of laws that we cannot do, which is breaking into another company."

Banafa said the companies' decision to disclose the incidents publicly is a positive step aimed at building trust, and predicted the disclosures will accelerate pressure from government and industry to build in safeguards from the start of AI development rather than after models are deployed.

"This is a warning shot for everybody in the industry," Banafa said. "Other companies are looking back at their books now to see if their models did the same thing."

Asked about accountability, Banafa said the situation represents new legal territory, since a human employee who accessed another company's systems without authorization would likely face legal consequences.

"There will be new rules, and I'm not going to be surprised if a company in the future will sue one of those companies because their AI went outside and grabbed the information from a certain company," Banafa said. "We're getting to this gray area where — who's responsible?"

Banafa said addressing the issue will require action from government, companies and users together.

"Regulation never curbed innovation if it's done right," Banafa said. "I mean, the seat belt never stop the auto industry from advancing. The cyber attacks never stop the internet from evolve and becoming better. The same thing with the AI safety. We need something reasonable, something which is accepted by three parties: the companies, and the government, and us as users, will make it safer. The companies cannot police themselves. We tried that with social media before."

Banafa also addressed a separate report from The New York Times this week on possible hacks targeting water utilities in seven states. He said water systems are considered a "soft target" for cyberattacks because they are not typically funded for the same level of cybersecurity as sectors like banking or technology.

He said the reported intrusions affected roughly 30 facilities in Minnesota and nine in Michigan, mostly in smaller communities.

"Thank God they do not do anything about poisoning the water," Banafa said, adding that the FBI's involvement is a positive sign that the intrusions are being addressed.

Technology