Note: the following content is not necessarily endorsed by the experts listed in the AISCC database.

Introduction

Interest is surging in AI regulation as systems rapidly advance in capabilities and deployment. While governments have been considering how to regulate AI for some time (e.g. the US AI Bill of Rights and the EU AI Act), the rapid advance in AI capabilities and deployment (as seen by GPT-4, Bard, and Claude) has led to renewed regulatory interest. Policymakers are looking to develop frameworks that can mitigate risks posed by advanced AI systems, while still allowing the public to benefit from innovation in the field.

Regulating AI faces a number of challenges in balancing various stakeholders, tradeoffs, and the quick pace of development. Still, in recent months, experts have begun to converge on proposals that, if enacted, could significantly reduce risks.

Why regulate?

Advanced AI systems hold tremendous promise in areas like healthcare, climate change mitigation, and more. However, they also pose significant risks. Dangerous capabilities, such as the ability to design new bioweapons or aid in cyberattacks, could prove to be extremely harmful unless properly mitigated. Advanced AI systems can also perpetuate racial or gender bias, potentially violate artists’ copyright, and lead to a proliferation of harmful material (such as child sexual abuse material). Some also worry that future systems could evade human control, as the challenge of reliably controlling the intentions and actions of advanced AI systems is an unsolved technical research problem. As one paper notes, such capabilities “can arise unpredictably and undetected” and, once deployed, preventing harm will be difficult.

Without mitigating these risks, it will be impossible to harness the benefits of AI. Furthermore, regulation is needed to create a stable economic environment for AI: without liability frameworks and clarity on IP ownership, some companies may avoid using AI altogether.¹ Overall, regulation is necessary to curb dangers while allowing society to benefit from AI.

A tiered approach to regulation

Many experts believe a blanket regulatory framework would be overly burdensome on lower-risk AI systems.² Instead, a tiered model is often advocated, where regulation increases with the extent to which a system is capable of causing harm.³ The most stringent regulation would therefore only apply to a very small number of models (those with the potential for widespread impacts or abuse) and their developers. The aim is that this would allow us to capture most of the societal benefits of AI while avoiding the downsides.

For brevity’s sake, this briefing will primarily focus on these highest-risk systems, often referred to as “frontier models”.⁴

Standards, evaluations and licensing

A common proposal is establishing a framework of requirements that frontier systems must meet before deployment.

This framework, which has been elaborated on at length by the Institute for Advanced Study’s AI Policy and Governance Working Group and the Centre for Governance of AI et al looks something like:

Governments, working with academia and industry, draw up a set of requirements for frontier models. This might include things like:
- Risk assessments for dangerous capabilities.
- Mandatory evaluations by a third-party auditor.
- A tiered deployment framework that restricts deployment until certain standards can be met with a high level of confidence.
Governments ensure compliance with these standards, either post-development or beforehand (via a licensing system).

In a survey of 51 AI governance experts, 98% of respondents “somewhat or strongly agreed” that AGI labs should conduct pre-deployment risk assessments, dangerous capabilities evaluations, third-party model audits and red teaming.

What standards could look like

There is much discussion of what standards organisations training frontier AI models should meet.

Ideas under consideration include:

Risk assessments and evaluations of model capabilities — before training, during training, before deployment and after deployment.
- Capabilities that could be assessed for include:
  - the ability to design bioweapons
  - the ability for the system to shape and manipulate people’s beliefs, including persuading people to believe false information
  - the ability for the system to replicate itself ⁵
  - whether the system has situational awareness (e.g. does it know it is being trained or evaluated)⁶
Evaluations of whether models reliably do what their developer or user intends, including robustness to adversarial attempts to override guardrails, or attempts by the system to deceive its evaluators.⁷
- This could also include evaluations for e.g. a system’s ideological or racial bias.
Risk assessments and evaluations must be done by an independent third-party auditor.
Standardised protocols for how systems are deployed, given their assessed risk level.
Regular re-assessments of models as their capabilities are enhanced post-deployment (through e.g. prompt engineering or fine-tuning).
Transparency into how AI systems are trained, including the datasets used to train systems.
Explainability into why AI systems make certain decisions.
Internal governance practices, such as whistleblower protection and oversight committees.
Mandatory incident reporting when systems do display concerning behaviours, including details of the training approach used to develop the risky system.⁸
Mandatory security practices to protect frontier AI models from leaking or being stolen.

This framework looks likely to become the basis for legislation. Most notably, it closely resembles the voluntary commitments that major AI labs recently made to the White House: labs have committed to testing for dangerous capabilities (including, notably, the capacity for models to “self-replicate”), and to implement various security practices (such as limiting access to model weights). This builds on existing self-regulation from labs (e.g. OpenAI’s GPT-4 and Anthropic’s Claude underwent an extensive testing process).

Elsewhere, regulators are already drawing up AI standards (such as the National Institute for Standards and Technology’s AI Risk Management Framework), though more work is needed on frontier-level standards. The Algorithmic Accountability Act, meanwhile, calls for companies to evaluate AI systems for bias. The legislative framework proposed by Senate Majority Leader Chuck Schumer also relies heavily on AI models meeting a set of standards. Schumer is particularly focused on “explainability”, which, given our present lack of understanding of the internals of advanced AI systems, is one of the most difficult standards for current systems to meet. Evaluating systems for dangerous capabilities is also picking up steam: legislation proposed by Sens. Ted Budd and Ed Markey aims to prevent AI being used to develop bioweapons.

How to enforce standards

Given industry support for safety standards, it seems likely they will initially be employed voluntarily. However, government regulation will likely be needed to enforce such standards in the long term.

Governments could ensure compliance through post-deployment enforcement: if developers release models that do not meet safety standards, they can be fined or otherwise penalised. This, however, has the risk of action being taken too late: if regulators find out post-deployment that a system can autonomously replicate and spread through computer systems via hacking, the damage has already been done.

A more proactive approach would be to require licences to deploy and/or develop frontier systems, a framework similar to pharmaceutical and aviation regulation. Developers would apply for permission to develop or release their models, demonstrating in advance that the various safety standards have been and will be met.

Licensing is growing in popularity. In the US, Andrew Tutt proposed something similar back in 2017, while Sens. Josh Hawley and Richard Blumenthal have more recently emphasised the need for this approach. Other experts have also called for a licensing regime, including Jason Matheny, the CEO of RAND Corporation, and Gary Marcus, an AI professor at NYU. The UK’s Labour Party has also endorsed these ideas.

To help ensure only licensed activity is taking place, governments will need to monitor and enforce who is actually developing and deploying frontier models. As extremely large amounts of computing power are currently required to develop such models, one promising route is “compute governance” — monitoring of, and restricting access to, the computing power needed to build frontier systems, and boosting access for socially beneficial uses of AI. This could be done at either the chip or datacenter level. A version of these restrictions are already in place: through semiconductor export controls, the US, Netherlands and Japan have restricted China’s access to the advanced chips needed to train frontier models. Recent reports suggest these controls may soon be strengthened and expanded to Chinese companies’ access to cloud computing. These restrictions allow Western governments to proceed with regulation without China unilaterally pushing ahead.

Procurement rules

Governments are not only regulators of technology — they are also among the largest customers. They can use this leverage to encourage AI developers to increase the safety and reliability of their systems. For instance, governments could refuse to use AI systems unless they are demonstrably non-racially-biased, or robust to adversarial attacks.

Increased visibility into AI development

Much of the above can only be done if regulators have access to information. Regulators need increased transparency about how AI models are trained and deployed, via something like the “model card”, pioneered by Margaret Mitchell et al.

Additionally, experts have called for AI labs to either voluntarily or mandatorily submit disclosures to regulators. Labs could also provide regulators with information on the status of model development and deployment, including planned training runs or compute use.

Some of this information might also be provided to third-party researchers (such as academic institutions), who can provide a further check. This will, however, need to be balanced with privacy, proliferation-risks, and commercial interests.⁹

Transparency is also part of the White House-organised voluntary commitments: labs have agreed to “publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use, including discussion of societal risks, such as effects on fairness and bias”.

Clarity around legal questions

To create a stable economic environment for AI, governments need to address and clarify legal questions surrounding its development and use. Decisions must be made regarding who, if anyone, is liable for harms perpetrated by AI models. (Sens. Hawley and Blumenthal want to make AI developers liable, exempting them from Section 230 protections).

Governments must also clarify how copyright laws apply to AI systems. Rulings issued by the US Copyright Office suggest AI-generated products can’t be copyrighted, but more clarity is needed. Privacy is another area where more clarity is needed, given the potential use of personal data in training models.

People have also called for clarity on whether antitrust measures will apply to labs that coordinate or share information with one another to increase industry safety. In the absence of clarity, labs may be reluctant to share safety insights, increasing the risk of accidents.

Funding for safety research

Many have also called for governments to play an active role in solving the many technical problems in AI safety, such as improving our understanding of how advanced systems work (interpretability) and getting systems to better follow their developers’ and users’ intentions (alignment).¹⁰ Solving these difficult technical problems may require the kind of large-scale research project that governments have previously executed (e.g. the Apollo Project, Operation Warp Speed).

The UK’s Foundation Model Taskforce has already committed £100m to AI safety research, while the US National Science Foundation is spending $20m on various AI safety projects. Funding for the US National AI Research Resource, meanwhile, is currently being discussed. Others have called for increased funding for NIST so it can build better AI testing environments.

As access to large amounts of computing power and frontier systems is often needed to conduct useful safety research, some have also called for governments to improve academia’s access to these resources. This could take place through the development of national supercomputers¹¹, by purchasing cloud compute credits for academia, or by partnering with AI labs to ensure controlled academic access to frontier systems.

International governance

The risks of AI models, if they materialise, will affect everyone around the world. In addition to domestic regulation, therefore, international governance is likely needed.

This short briefing cannot do justice to the myriad ideas for international governance. In a paper titled “International Institutions for Advanced AI”, authors from academia, labs and think-tanks propose four ideas:

A “Commission on Frontier AI”, similar to the International Panel on Climate Change, to establish a scientific consensus on the opportunities and risks posed by advanced AI.
An “Advanced AI Governance Organisation”, similar to the International Civil Aviation Organisation or International Atomic Energy Agency, to set international governance norms and standards; as well as potentially monitoring compliance.
A “Frontier AI Collaborative”, similar to Gavi, that ensures all countries and communities have access to the benefits of advanced AI.
An “AI Safety Project”, similar to CERN, which brings together international researchers to advance AI safety research.

“IAEA for AI” and “CERN for AI” have received significant attention from industry, government and academia.¹² Notably, the United Nations has expressed interest in an international AI “watchdog”, with Secretary-General Antonio Guterres saying he is “favourable to the idea that we could have an artificial intelligence agency … inspired by what the international agency of atomic energy is today”.

Challenges to governance

Few experts would claim to have fully fleshed-out governance proposals yet. Significant work is needed to develop the ideas outlined above.

As policymakers do so, there are numerous considerations that must be kept in mind:

How to govern in a way that does not excessively stifle innovation
How to avoid regulatory capture by leading AI labs
How to avoid concentration of power among AI labs
How to avoid exacerbating economic inequality and widespread unemployment
How to define “frontier” models
How to monitor access to compute without excessive surveillance
When more stringent regulation should kick in
How to avoid regulatory flight
How to ensure international cooperation that includes both the US and China

Such challenges will require careful thought, and the input of many different stakeholders and communities.

No perfect proposals exist yet. As the AI Policy and Governance Working Group at the Institute for Advanced Study said in their recent recommendation report, “no single accountability intervention or organisation will be fully effective on its own”. Instead, a patchwork of interventions, both domestic and international, will likely be needed to reduce the risks of advanced AI systems.

Footnotes

See e.g. Valve Software, who have banned AI-generated content from their Steam games store until there is more clarity around copyright ownership. ↩︎
See e.g. pg. 20 of this paper: “imposing [licensing requirements] on present-day AI systems could potentially create excessive regulatory burdens for AI developers which are not commensurate with the severity and scale of risks posed.” ↩︎
See points 2 and 4 here. ↩︎
In Frontier AI Regulation, the authors define frontier models as “highly capable foundation models”, with foundation models defined as “models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks”. ↩︎
See Update on ARC’s recent eval efforts for more on this. ↩︎
See Model evaluation for extreme risks for a more comprehensive list of dangerous capabilities. ↩︎
Note that getting reliable versions of such evaluations is expected to be difficult, as we can not yet reliably ensure a model is aligned. ↩︎
See Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (Brundage et al, 2020) ↩︎
See Shevlane (2022) for an extended discussion on how AI scientists can share their work. ↩︎
e.g. recommendation 3 from the Federation of American Scientists’ AI policy ideas, or the Foundation for American Innovation’s proposal for a “Manhattan Project for AI Safety”. ↩︎
e.g. “Building Compute Capacity in a Responsible Manner” in the Tony Blair Institute for Global Change’s paper on AI. ↩︎
See Rishi Sunak on CERN for AI and OpenAI executives on IAEA for AI. ↩︎

AI Safety Communications Centre

Proposals for AI regulation