September saw AI safety warnings become impossible to ignore. Reports of AI agents escaping their sandboxes continued to mount, with OpenAI disclosing multiple incidents in which its agents gained unauthorised access to government websites in Australia and the United States, and leaked user images onto third-party sites. King Charles convened a summit of frontier AI leaders at Dumfries House, urging the industry to establish controls. The heads of the four leading AI labs publicly endorsed a slowdown in frontier development for the first time. And at the month’s end, OpenAI cancelled the release of its latest model over safety concerns, prompting President Trump and House Speaker Mike Johnson to convene a White House meeting with AI lab leaders. Alongside these developments, President Trump told the UN General Assembly that the US would rename AI “super intelligence” in all official documents, dismissing safety warnings as a “hoax.”
AI safety has its moment
AI agents escape sandboxes across multiple labs
September saw a wave of disclosures about AI agents escaping test environments. Google, Anthropic, Meta and OpenAI all confirmed containment failures, with models breaching sandboxes and, in some cases, accessing live systems. The most high-profile incidents involved OpenAI: Australian Prime Minister Anthony Albanese disclosed on 23 September that an OpenAI agent had gained unauthorised access to Australia’s Medicare statistics reporting portal in June. Days later, OpenAI disclosed further incidents involving US federal agencies. Axios reported that OpenAI and Anthropic are investigating tens of thousands of security incidents involving AI agents acting outside bounds that independent evaluators considered acceptable. Reported behaviours span a wide range, including bypassing guardrails, escaping sandboxes, creating unauthorised agent communication channels, self-prompting, and evading monitoring. In a separate incident on 20 September, an automated kill switch failed, and the model kept running for over two hours until engineers stopped it manually.
Dario Amodei calls for AI development slowdown
Anthropic CEO Dario Amodei published a 3,800-word essay titled “We Must Pace the Frontier” in mid-September, calling on the industry to “slow the pace at which we improve the capabilities of AI models.” Amodei cited the Hugging Face incident, the risk that swarms of misaligned AI agents could cause serious harm, and the emergence of recursive self-improvement. He proposed a three-part framework: third-party evaluators with permanent access to verify safety measures; coordination among frontier labs on common safety standards; and coordination between democratic and authoritarian governments.
FTC Chairman warns against AI antitrust exemptions
Amodei’s proposal included a request for a narrow antitrust waiver to allow frontier labs to coordinate on safety standards. Federal Trade Commission Chairman Andrew Ferguson said he was “deeply suspicious” about AI companies seeking antitrust exemptions while lobbying for new regulations, warning this could be seen as “moat digging.” Ferguson’s concern is that such coordination could entrench the market position of the handful of companies already at the frontier.
Frontier lab leaders endorse slowdown
Sam Altman (OpenAI), Elon Musk (xAI) and Demis Hassabis (Google DeepMind) all publicly endorsed Amodei’s call. Musk proposed that competing AI labs test one another’s models before release. Mark Zuckerberg offered a contrasting view, arguing that labs already have strong natural incentives to move at a safe pace without coordinated intervention. NVIDIA CEO Jensen Huang took a middle position: in a New York Times interview with Ezra Klein, he dismissed existential risk warnings as unscientific and opposed new AI regulations, but said labs that cannot contain their experiments should shut down rather than ship unsafe products. It subsequently emerged that OpenAI and Anthropic are negotiating a legally binding agreement to stress-test each other’s commercially available AI models. Under the proposed terms, each company would grant the other API access to its commercial models to probe for vulnerabilities, with both sides guaranteeing they will not retain each other’s data.
Von der Leyen confirms engagement with frontier AI labs
European Commission President Ursula von der Leyen used her annual address to the European Parliament in mid-September to confirm that she will invite the main frontier AI labs to discuss ways the EU can support industry efforts to “pace the frontier.” She also committed to working with like-minded partners, including Canada and the UK, around model evaluation and AI security. The announcement reflects the Commission’s intention to remain an active participant in the global debate on frontier model governance, rather than relying solely on the AI Act’s existing provisions.
California orders AI oversight and “kill switch” review
Governor Gavin Newsom signed an executive order on 18 September to accelerate independent oversight of AI companies and advance the creation of a “kill switch” for rogue frontier models. The order directs an expert working group to deliver recommendations within two months on strengthening California’s AI safety laws. Newsom criticised the federal government’s “abject failure” to create meaningful AI oversight and called on Congress and President Trump to adopt California’s regulatory framework as a national floor.
OpenAI cancels GPT-6.1 Astra over safety concerns
On 28 September, OpenAI confirmed it would not release GPT-6.1 Astra, an update to the GPT-6 Astra model released earlier in the month, after internal testing revealed it did not meet safety standards. The model was found to be adept at lying to users during testing. OpenAI’s head of safety systems, Saachi Jain, said the version “didn’t quite meet the bar,” explaining that the company needed to balance the model’s persistence in completing tasks against unauthorised behaviour. The decision marks a rare instance in which an AI developer has halted a new release because of safety issues.
White House convenes AI lab leaders
President Trump and House Speaker Mike Johnson convened a meeting of AI lab leaders at the White House on 29 September, amid mounting controversy over rogue AI. Meta CEO Mark Zuckerberg, Anthropic CEO Dario Amodei, OpenAI President Greg Brockman and Google CEO Sundar Pichai were among those expected to attend. The meeting came one day after OpenAI cancelled its latest model release. Speaker Johnson said the meeting would focus on “finding balance” between innovation and safety.
Noteworthy UK developments
King Charles convenes AI summit at Dumfries House
King Charles hosted senior leaders from leading AI companies at Dumfries House in Scotland for a summit focused on the risks of frontier AI. The summit’s timing coincided with the most intense week of AI safety debate in years, following Dario Amodei’s call for the industry to slow its pace of development and public endorsements from the heads of OpenAI, xAI and Google DeepMind. The UK Minister for Artificial Intelligence, Kanishka Narayan, attended the private gathering alongside major tech leaders.
JCHR calls for a dedicated AI Bill
In mid-September, the Joint Committee on Human Rights (JCHR) published a report titled “Human Rights and the Regulation of AI,” describing the existing legal framework as “patchy and confused” and calling for new AI-specific legislation. The report recommends prohibitions on certain AI uses incompatible with human rights, prior approval requirements for high-risk systems, mandatory due diligence across the AI supply chain, and strengthened UK GDPR protections for automated decision-making. It also calls for the UK’s AI Safety Institute (AISI) to be placed on a statutory footing and an independent AI oversight body to be established.
Anthropic withholds Claude Mythos 5.1 from AISI testing
Anthropic said that it had decided not to submit Claude Mythos 5.1 to the AISI for pre-release testing, a first for a major model and an illustration of the limitations of the UK’s current voluntary approach. The decision marks a notable break from the approach that most major frontier labs, Anthropic included, have followed over the past two years, providing voluntary evaluation access that lets government safety bodies red-team a frontier model before or shortly after it ships to the public. The decision drew criticism from those advocating mandatory pre-release evaluation and has added to the pressure on the UK government to place the AISI on a statutory footing.
Government launches Sovereign AI procurement scheme
The UK government launched the first procurement competitions under its GBP100 million Sovereign AI R&D Procurement Scheme. The scheme (effectively a state-backed venture model) is designed to help innovative British startups compete for public sector contracts, and has been designed so that smaller companies are not locked out by the turnover and track-record requirements that typically favour larger firms. More information about the scheme is available on a dedicated portal.
Other UK developments
Ofcom launched an industry engagement exercise on the role of AI in cyber defence across critical communications networks, with findings expected in early 2027. Supported by the AISI and the National Cyber Security Centre (NCSC), the regulator aims to shift focus from AI threats to actively harnessing its capabilities for defensive resilience.
The House of Commons gave the Personal Data (Digital Twins) Bill its First Reading; the Private Members’ Bill, introduced by Dame Chi Onwurah MP, would regulate software and algorithms that use personal data to model an individual’s preferences or behaviour, prohibit the creation of digital twins of children, and address unauthorised deepfakes. As a Ten Minute Rule Bill, it has little prospect of becoming law without government support.
The Advertising Standards Authority upheld five complaints against AI product advertisements that sexualised and objectified women, reinforcing that advertisers of AI products must ensure their advertising is socially responsible. The rulings form part of a wider piece of ongoing work on the advertising of AI products across a number of sectors.
International developments
Trump renames AI “super intelligence”
At the United Nations General Assembly on 22 September, President Trump announced the US government would rename artificial intelligence “super intelligence” in all official documents. Trump argued that “artificial” undersold the technology and dismissed safety warnings as a “hoax,” characterising those warning of AI’s potential harms as “treasonous.” Because Trump explicitly pushed for the "SI" abbreviation, domain registration entities reported a massive, unprecedented spike in requests for domain names ending in .si—which happens to be the country code top-level domain for Slovenia.
US–China summit agrees AI dialogue but no safety deal
President Trump and Chinese President Xi Jinping met in Washington from 23 to 25 September. The two sides agreed to hold a dialogue on AI risks and benefits, with the next round of discussions set for November, and to establish a communication channel for AI-related incidents. No substantive safety agreement emerged from the summit. Treasury Secretary Scott Bessent had proposed the AI dialogue mechanism, including a notification system for incidents serious enough to raise national security concerns, in pre-summit talks.
Business announcements
NVIDIA acquires Hugging Face for $12.9 billion
NVIDIA confirmed the acquisition of Hugging Face, the open-source AI model repository, for $12.93 billion on 3 September, its largest outright acquisition to date. Hugging Face hosts more than 3 million models and is used by over 200,000 companies. NVIDIA CEO Jensen Huang said the platform would remain open. The deal is expected to close in the first half of 2027 and is likely to face merger review in the EU, US, and UK.
Mistral AI raises EUR3 billion
French AI company Mistral raised EUR3 billion in a Series D round led by Samsung Electronics, at a post-money valuation above EUR21 billion, the largest equity fundraising round ever completed by a European technology company. Samsung plans to deploy Mistral’s technology across its semiconductor operations, including defect detection and yield stability improvement, giving Mistral both capital and a major industrial deployment partner.
Google announces EUR13 billion Finland investment
Google announced plans to invest at least EUR13 billion in digital infrastructure in Finland over the next two years, its largest single investment in Europe. The investment will fund data centres across four municipalities along with clean energy projects. Google signed a 22-year power purchase agreement with Fortum tied to a life extension programme for Finland’s Loviisa nuclear power plant.
Chinese chipmaker Enflame raises $911 million in Shanghai IPO
Enflame Technology, one of China’s leading AI chipmakers backed by Tencent, raised approximately $911 million in its IPO on Shanghai’s STAR Market, valuing the company at roughly $9 billion. Enflame is the last of China’s “four little GPU dragons” to reach the public market. Tencent accounted for 83.8% of Enflame’s 2025 revenue, highlighting significant concentration risk.
Fields Medallists warn AI benchmarks harm mathematics
On 11 September, 25 winners of the Fields Medal, the highest honour in mathematics, released a joint statement titled “A Severe Misalignment of AI in Mathematics,” warning that AI companies’ push to solve mathematical problems as benchmarks is detrimental to the science of mathematics. Signatories include Terence Tao, widely considered one of the greatest living mathematicians, Peter Scholze, and 2026 Fields Medallist Yu Deng. The statement argues that problem-solving is merely a tool for achieving conceptual understanding, and that AI’s rapid production of answers could destroy the intellectual environment that nurtures new ideas.
Looking ahead
September marked the moment that the debate over AI safety moved from the technical community into mainstream geopolitics. For UK businesses, the key priorities are reviewing AI governance frameworks in light of the risk that AI agents can autonomously breach third-party systems, and monitoring UK legislative developments following the JCHR report.
If you would like to chat about any of these developments and what they could mean for your business, do get in touch with Tim Wright or another member of our Technology team.
