Anthropic calls for $15 million in funding to boost the government’s AI risk assessment work

The AI research company Anthropic is calling for $15 million in additional funding for the National Institute of Standards and Technology (NIST), to help the agency expand its work on a framework to evaluate the capabilities, limitations, and risks of large language models (LLMs).



“We asked ourselves the question: What would it look like if this agency had more people, and we think the answer is that it would be able to do more and better work,” Anthropic cofounder Jack Clark tells Fast Company. Anthropic also says NIST needs 22 additional staff positions for its AI program.



NIST, which is part of the Commerce Department, has already been working hard on evaluating the risk of AI systems. In late January the agency released the AI Risk Management Framework, a set of guidelines that can be used by organizations to manage risk during the development of new AI systems.



In late March it launched the Trustworthy and Responsible AI Resource Center, which offers tools to help AI developers implement the risk management framework. Previously, NIST created a framework for building and assessing cybersecurity systems that was widely adopted by companies in numerous industries. It also created a framework for testing facial recognition systems, which use a different form of AI than ChatGPT-style chatbots, generative AI systems powered by large language models.



“You would imagine that they’re going to do it for some of these generative AI systems as well,” Clark says, “but other issues might include how truthful are generative AI systems over certain topic areas? How likely are they to make up facts or cite sources or other things?”



Anthropic’s call comes at an inflection point when artificial intelligence will soon play a major role in business and personal life. Momentum in the development of LLMs continues to increase despite the fact that the models often perform in unpredictable ways, and their outputs cannot be linked to the specific training data that drove them. Further, AI companies can’t fully anticipate the potential harms of the technology during the development process. To the extent that it can, the AI community lacks consensus on how to access the risk.



Meanwhile, many in government would like to get in front of the paradigm shift to protect people from AI’s potential harms, after failing to protect consumers, including children, from the harms of social networks. But any hope of effective regulation, Anthropic believes, depends on reliable measurement and risk analysis. Lawmakers naturally look for objective and nonpartisan sources of information when making policy decisions. They look to the Congressional Budget Office, for example, to measure the likely effect of fiscal policies.



“One thing that would help in Washington is having more data about AI coming from within government, rather than exclusively from outside,” says Clark, who formerly worked in public policy (government affairs) at OpenAI. “NIST could be a place that talks to third parties [tech companies and tech policy groups], but also makes its own considerations and its own judgments, and provides it as a resource for government.”



Tech companies have long been granted the privilege of self-regulation, but many in government believe the industry has abused that privilege and needs greater government scrutiny. Some lawmakers may sense that the stakes are too high with AI to leave to leave oversight to the private sector. Note, however, that the risk assessment frameworks NIST has so far created are completely voluntary and nonbinding.



At the same time, Clark says, the tech industry could benefit greatly from an open, data-sharing relationship with NIST. He points to the fact that AI development companies were very open with NIST when the agency asked for their input during creation of its risk management framework.



“That told us [that] there’s already tacit buy-in about NIST,” he says. “I think companies like it because they see NIST as a place where you can send them information about the evaluations and measurements that you’re doing, and NIST is going to make its own determination about which of those it thinks are especially good.”



“It’s actually a better and healthier place to be developing some of these measures and evaluations,” he continues. “With a government partner where you can then kind of send them and the government partner can scale them up and criticize them and develop them.”



Anthropic was founded in 2021 by former OpenAI VP of research Dario Amodei and his sister Daniela, as well as a group of OpenAI executives, with the intent of developing “large-scale AI systems that are steerable, interpretable, and robust.”