Anthropic has filed its initial public offering prospectus, dedicating 94 pages to risk factors that could threaten both the company’s business and humanity itself, according to Wired.
The filing identifies recursive self-improvement as a central concern. RSI describes AI systems autonomously developing new AI models without human involvement. “Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models,” the prospectus states.
Anthropic warns that “sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity.” The phrase “existential risks” appears three times in the filing, citing concerns from critics, regulators and inside Anthropic itself.
The prospectus repeatedly makes the point that RSI could lead to AI models that develop faster than the company’s safeguards, and that the company may not even realize it has lost that race until it’s too late. While AI already plays a role in model development, the doomsday scenario involves humans being entirely removed from the development loop. According to Wired, at least one lab has used something close to RSI to speed up development on a model that punches above its weight.