Feeling the Crisis

Strong policy change happens most effectively when the attributed problem is felt enough to warrant the perceived costs of the proposed solution. Many relevant precedents to AI governance can be viewed through this lens:

• What led to the rapid establishment of the IAEA and NPT, in spite of the costs endured by doing so? It’s because the effects of nuclear war and threat of mutually assured destruction had just been felt by everyone, so they had better make sure it never happens again.1

• How did we enact such strong enforcement measures for mitigating the COVID-19 pandemic so rapidly? It’s because decisionmakers felt the rapid spread, the climbing death toll, the general feeling of uncertainty surrounding how bad it was going to be, and the established experts sounding the alarm on what ought to be done, enough to warrant using their authority to invoke emergency measures taken. Why did we underprioritize stronger pandemic prevention and even roll-back some pandemic response measures afterward? It’s because people felt the negative impacts of the lockdown mandates far more than the pandemic itself or the impacts counterfactually not experienced as a result, leaving many people overcorrecting towards general opposition to pandemic-response measures.

• Why is there still no law mandating nucleic acid screening, so as to ensure someone trying to order a virus from a nucleic acid synthesis company would be caught and stopped before they could do so? Seriously, this is essentially unanimously supported, has had advocacy for decades, including by the industry themselves and especially recently, and yet the bill that could solve this has been waiting in committee without markups for just over 7 months now after being introduced. It’s because Congress is slow by design, devoting time to a particular bill or issue costs time spent on others, and plenty of other bills feel like more of a priority. There have been no recent attempts by bioterrorists to use this to their advantage, after all.

• Why did the thalidomide crisis prompt the safety and efficacy requirements on the development of new drugs? And why do these persist in spite of slowing down drug development far more than what would have been enough to prevent the thalidomide crisis? It’s because the thalidomide crisis was felt by everyone from the scandal it produced, and you never feel the downside of medicine that could have helped you counterfactually not being available earlier. At least, not in such a way that leads you to attribute it to the drug approval process needing reformed.

• Where has the environmentalist movement succeeded, in spite of its worst risks, like AI, also being hypothetical long-term events never felt? Especially in spite of the group that reaps the concentrated benefits from not complying with regulation investing far more resources to influence political outcomes in their favor than the general public who only experiences its diffuse costs? It’s where people saw and felt its near-term consequences the most: the Great Smog of London prompting the UK’s Clean Air Act, the Cuyahoga River fire prompting the expansion of the Clean Water Act and accelerating the creation of the EPA, Love Canal’s contamination and resulting adverse health effects prompting Superfund, etc.

• Well then how come the Montreal Protocol succeeded? They started reversing the depletion of the ozone layer well before its adverse effects were felt by many people, and it suffered from the concentrated benefits versus diffuse costs problem, and it’s a binding international agreement which is much more difficult to establish, so it’s unusually difficult for many of the same reasons a binding international AI treaty would be! It’s partially because advocates leaned into the more immediate “the ozone hole causes cancer” messaging than “the ozone hole will gradually make the Earth less habitable”, partially because environmentalism was already a much larger political movement due to other factors, but in this case it was actually primarily because the most active lobbyist group found harmless alternatives to the substances depleting the ozone layer that weren’t particularly costly to switch to.

Feeling the AGI

• Why did the perceived jailbreak to access Mythos’ advanced cyber capabilities prompt its export controls? Because the government had already felt its capabilities from the reports surrounding it, and wanted to be especially cautious to prevent foreign actors taking advantage of it.

• Why did the OpenAI HuggingFace security incident lead Trump to tell reporters the White House was looking at ways to balance controls for AI with not coming second to China? Why did several2 members of Congress use this event and related reports to point to the need for mandated transparency? Because the news coverage, Sam Altman, and over a thousand frontier AI lab employees helped them feel the seriousness of the event.

• Why has the White House otherwise been so averse to AI regulation, citing the need to beat China in the AI race? Because that’s what all the loudest voices have been telling them (and because this administration tends to be more skeptical of economic regulation in general, which could be a good heuristic under normal circumstances).

• Why does the general public seem far more concerned by data centers? Because regardless of how much they use AI themselves, or indirectly benefit from companies whose products they buy using it, they only feel the giant ugly buildings being built nearby, and they hear that it’s using up all their water. One AI company’s agent hacking another AI company sounds like their problem.

• Had the victim of the HuggingFace incident instead been any of the vast majority of institutions that would have sued OpenAI in response, prompting the heavy news coverage and legal ruling that would have followed? Had it been a government agency? An airport? A hospital? Any other piece of digital infrastructure providing services frequently used by a large portion of the general public and viewed as important? Had it tangibly inconvenienced the people who used the service, even if only in some minor way? Had they really felt it? Raising transparency, incident reporting, control, and liability standards for AI labs would be the least of the resulting demands made.

This, the several similar reports that shortly followed, and the fact that OpenAI has already continued development suggests we’ll likely be getting more incidents of this kind3. They’re making it easy for us; all we have to do is not drop the ball. But as “The current bottleneck is political will, not research” recently illustrated (which I strongly urge you to read4 if you haven’t already), the ball is being dropped nowhere harder than here.

The main reason seems to be that passing legislation takes a lot of sustained effort, and there’s no crisis or warning-shot to make it feel urgent enough for Congress to prioritize. The relevant bill here is S.3741, introduced January 29, 2026, which would solve much of the problem, at least on a national scale. This post by Sophie Kim goes into more detail, and provides additional commentary and proposed amendments. It’s come after decades of advocacy, most of the nucleic acid synthesis industry already voluntarily screening orders1, and being recommended in the Executive Office of the President’s July 2025 AI Action Plan for institutions receiving federal funding. It’s also worth noting that the open letter that called on Congress to pass this, signed by dozens of leading experts in several relevant fields (which helped to indicate the cost on innovation would be minimal), was published June 3, 2026, and that many members just weren’t aware that open-source AI tools can already design new dangerous proteins prior to this. Per Fortune:

While the bill slowly moves its way through Congress, Josh Wentzel, a senior fellow at the Foundation for American Innovation, told Fortune that the letter was a good opportunity to show lawmakers that the AI industry and companies who sell synthetic DNA and RNA were equally concerned about the issue.

“This is bipartisan, concrete, achievable, and noncontroversial,” Wentzel said, adding he hopes now that Congress sees these parties are aligned, it can move forward with passing the Biosecurity Modernization and Innovation Act. “It’s a goal among many national security experts and, crucially, something the nucleic acid synthesis industry itself has called for.”

So, the open letter does seem to be moving the bill faster than it otherwise would have. But in spite of all of this, and strong media coverage of the letter, and advocacy from the think tanks that co-organized the letter, the bill is still waiting in committee without undergoing any markups two months later. The bill is practically certain to pass eventually, but short of some event that would make it a higher priority, it’s likely going to take several more months. For reference, the much more mainstream Epstein Files Transparency Act took about 5 months to pass after being introduced. Congress is really that slow. I knew they had a reputation for it, but it’s disheartening to see it for myself. This lends credence to my prior theory that not much is going to concretely get done in AI governance until the effects are felt more, which I intend to investigate next.

Generally, AI labs should (and most do) put their models under extensive safety testing before deploying them to prevent misuse, scheming, and other dangerous behaviors. This may include internal tests, red-teaming efforts by third-parties, etc. However, edge case safety vulnerabilities will likely slip through, and these can still cause damage. If any of the risks of AI systems from labs that implement strong safety measures in the first place (as some don’t) were to come to fruition, they would presumably be from safety risks missed by their internal testing. So, designing these well should probably be a high priority. Many people already try to find these and share them on Twitter for their own entertainment. To further incentivize finding these vulnerabilities, bug bounty-like programs, where labs offer financial reward for disclosing safety vulnerabilities, are well suited. These also have the added benefit of likely resulting in more realistic use-cases, as there may be more chances for users to discover safety vulnerabilities unintentionally, depending on the risk type. For instance, a user may find their agent has been victim to a novel prompt injection attack, and they identify the attack used. AI labs could then use this information to inform future safety measures to ensure safety vulnerabilities submitted as part of the program are not reproducible in future models. Or, perhaps to inform a decision to temporarily halt deployment of a model if the risk is particularly egregious, such as reproducible uplift of development of powerful CBRN weapons. Similar to this, Anthropic already delayed public release of Claude Mythos in response to the results of their internal testing of its cyberattack capabilities. Many people have already brought attention to this. Fortunately, Anthropic, OpenAI, and Google already offer programs like this. But unfortunately, these I see these as currently too narrow in scope and ambition.

OpenAI

OpenAI’s program mostly offers rewards for cases of agents causing material harm, including

• Prompt injections which result in harmful agentic behavior

• Agents bypassing intended permission limits

• Agents performing tool actions without proper user understanding/confirmation, or which are misleading to the user

• Agents misusing tools for actions which were supposed to be disallowed
  They provide some specific examples of what they’re looking for as well. They also include rewards for their models engaging in behavior that compromises OpenAI’s security, which is mostly unrelated to harms to users (although it could harm OpenAI themselves). They say they will accept reports for anything else that could lead to user harm which include remediation steps, but this standard is vague. They also explicitly state that cases of models generating disallowed content is out of scope due to being “complex and not addressable through traditional security fixes”, although they probably should accept such reports, at least for potentially dangerous information. Notably, they require issues to be “consistently reproducible”, and they “accept partial or probabilistic exploits if the result is still high impact, but the burden of proof is on the researcher to demonstrate it is not a one-off fluke”. On their blog post announcing it, they state that submitted safety vulnerabilities must be reproducible “at least 50% of the time”. This is far too high of a burden; the right vulnerability can cause substantial damage even if reproducible in only one scenario.
  Despite having started in July 2025, 6 vulnerabilities have ever been rewarded, and the average payout for the last 3 months is listed as $250 at the time of writing, which is their minimum payout. In the best case, this could just be due to them already having mostly robust systems (and I personally find this to be the most likely explanation). But it may also be due to not enough people being aware of this and attempting to find vulnerabilities, or worse, OpenAI trying to minimize what counts for payouts to minimize how much they spend. Bug bounty programs generally don’t have the best reputation when it comes to fairly compensating researchers, and this may be no exception. There’s no transparency for what has and hasn’t been accepted and why, so there’s no way to know.
  They also have a second safety bug bounty program for biorisk, where they offer $25,000 to find a true universal jailbreak that clears all five of their bio safety questions. However, it only applies to GPT-5.5 in Codex Desktop, it’s only open by application to “researchers with experience in AI red teaming, security, or biosecurity” and requires signing an NDA, and only lasts for about 3 months.

Anthropic

Anthropic’s program offers up to $35,000 for identifying novel jailbreak techniques in their models that can reveal detailed harmful information across a wide range of queries, specifically those which cause the model to answer a predetermined set of harmful biological questions. This is for the stated purpose of testing the robustness of their Constitutional Classifiers. Similar to OpenAI’s biorisk bug bounty, this also requires signing an NDA and being accepted by application or invite only, accepting only people with demonstrable relevant past experience. They also offer a regular security bug bounty program.
In the past, they’ve offered a handful of similar programs:

• In August 2024, they offered a program with up to $15,000 in rewards per novel universal jailbreak found which can provide detailed answers to harmful questions about CBRN weapons and cybersecurity.

• In May 2025, they offered a program with up to $25,000 in rewards per novel universal jailbreak which can elicit dangerous information about CBRN weapons. This was only open for about a week.

• In the same announcement for the above program, they also announced a public form for submitting universal jailbreaks on their frontier models at the time. In the submission form, they even provided a specific biological weapons question to get their models to answer for a submission to count as successful. However, they did not explicitly state they would be offering any financial rewards for successful submissions, just that they would “be in touch within 7 days” if the submission was found to be successful.

Google

Google’s program rewards finding:

• Cases where their AI systems take rogue actions with clear harmful security impact, e.g. from prompt injection

• Cases where their AI systems leak sensitive information of the user without their permission, e.g. sensitive emails

• Cases where their AI systems enable a convincing phishing attack which does not show the “user-generated content” warning

• “Model Theft Attacks that exfiltrate complete, detailed, and confidential model parameters”

• Cross-account context manipulation attacks which can result in a separate user’s AI system being manipulated by an attacker, e.g. a calendar invite sent to a victim which results in their AI taking actions unintended by the victim

• Their AI systems bypassing access controls which result in non-security-sensitive information being exfiltrated (to serve as a separate tier of rewards from exfiltration of security-sensitive information)

• Cross-user denial of service attacks for AI services

• Other security or abuse issues in their AI systems, at their own discretion
  They also explicitly state that they will not reward getting the model to generate policy-violating content (e.g. finding and successfully using jailbreak techniques), although they probably should for cases involving dangerous information. They also state finding cases where the model hallucinates are not eligible. Rewards are scaled based on how important the Google AI product they are found in is (e.g. AI safety vulnerabilities in Google search are given higher rewards than those found in NotebookLM), how high the vulnerability is ranked in their hierarchy (rogue actions and sensitive data exfiltration are the top priorities), and some other minor factors. The maximum reward, for a highest ranking AI vulnerability found in a highest priority Google AI product, is $20,000.

Suggestions for improvement

I suspect these programs could be substantially improved with:

• Easily accessible information about the program on the chatbot interface or API page, to increase the number of users aware of the program and thus the number who participate.

• Many examples across a variety of different kinds of safety risks of vulnerabilities which are eligible for reward and their reward amounts, as well as a variety of examples of vulnerabilities not eligible, with explanations for why each example is or is not eligible for their respective reward amounts if it isn’t already self-explanatory. If possible, this should include real examples of past submissions which have been accepted or rejected, published with as much detail as possible without potentially leading to further exploitation of the vulnerability or exposing any other sensitive information.

  • Transparency about past accepted and rejected vulnerabilities may also help in establishing a good reputation for AI labs fairly paying out for AI safety vulnerabilities, which could incentivize more people to attempt to find them.

• Just about any vulnerability that could cause a substantial amount of harm and could not reasonably be blamed on simple user error (e.g. using LLM-generated information that includes hallucinations in a critical scenario, as these AI systems usually have disclaimers that information may not be accurate), or be part of a larger attack which could, should be eligible for reward in all labs’ programs.

  • At the very least, all labs should offer financial rewards for examples of the model providing substantial uplift in CBRN weapons and cyberattacks, the model attempting to comply with assisting obviously very harmful actions even if the information is inaccurate, prompt injections, models in agentic environments taking actions which harm the user (where the user cannot reasonably be blamed) such as leaking their sensitive information without permission, models leaking sensitive information relevant to how they are set up, and any harmful scheming attempted by the model such as blackmailing its users. There are probably many more good candidates for this list I can’t come up with at the time of writing. Also, although there are disclaimers provided that AI responses may be inaccurate, as hallucination rates decrease to the point where internal testing no longer finds new cases, labs should offer rewards for these too. This is because, although the disclaimers may absolve AI companies of the responsibility, there will be increasingly high pressure to implement AI assistance in any area subject to competitive pressure as AI systems become more capable, including in cases where failure could result in serious harm, and AI companies ought to try to prevent this to the greatest extent they can.

• A much lower threshold for the percentage of cases in which the vulnerability is reproducible, if not only requiring one case. A terrorist group attempting to develop CBRN weapons would only need one successful jailbreak attempt to uplift their efforts (assuming the model is capable enough to provide substantially helpful information). Any of these instances have research value which ought to be rewarded.

• Higher maximum rewards for any especially egregious safety vulnerabilities, e.g. start-to-end uplift of individuals with no relevant expertise in creating CBRN weapons. These almost certainly wouldn’t apply to current models. Perhaps scaling of safety bug bounty rewards and which vulnerabilities are included for eligibility in safety bug bounties could be included in labs’ responsible scaling policies.

• A lower bar for entry to be accepted into private bug bounty programs, and/or a clear pathway to be accepted into them, for people who could contribute but don’t have past expertise to show for it (the bar should still be high enough to filter out most of the spam and low-quality participants). This could entail, for example, a test environment without sensitive information that would require signing an NDA where the user can try their strategies for finding AI safety vulnerabilities and explain their strategy and how it changes. It may also be feasible to use an LLM to grade the quality of the attempts and strategy instead of using humans to do so to save on resources, based on some predetermined set of criteria that graders would otherwise follow manually.

• Bug bounty programs which accept public submissions, to the greatest extent possible. To save on resources for checking each submission, LLM grading may also be feasible to use here.

• Internal studies conducted by AI labs to estimate the overall helpfulness of their AI safety bug bounty programs, as it may turn out this all makes negligible difference.

Other thoughts

Ensuring AI labs’ systems are secure in the normal cybersecurity sense should also be a top priority, and so their use of traditional security bug bounties is also highly valuable; I didn’t critique these because they are far more matured, and out of my expertise.
It may also be a good idea for organizations to put out bounties which prove the feasibility of hypothetical AI safety risks, such as models achieving autonomous replication and adaptation in controlled environments. This would be valuable for showing how they were achieved, and thus could inform how to mitigate them from being achieved in the future. (This is a very underdeveloped idea and may deserve its own post.)
It’s also worth considering this post‘s suggestion for organizations to provide large financial incentive for disclosing major AI safety risks happening privately, e.g. exposing a lab creating highly agentic AI systems without proper regard for safety.