AVOID DOOM'S ALAS* : A.I. COMPANIES ESSAY

WITHOUT a speed limit - both OpenAI and Anthropic are pursuing dangerous ''recursive self-improvement'' strategies, which enlist A.I. models themselves in design and train their own successors.

Many fear that recursive self-improvement will cause us to permanently lose control of A.I. One OpenAI researcher even characterized the approach as a '' runaway nuclear chain reaction '' that threatens everyone's survival.

But it's not necessary for the A.I. industry to wait for collective action. There are simple steps any A.I. company could take, today, to reduce the danger we face.

The first step would be to commit to meaningful incident disclosure, including defining what incidents warrant bringing in a third-party oversight.

Like in aviation, private companies should disclose not only actual breaches of safety, but near misses as well, lest we pay for each lesson with a tragedy. If A.I. companies continue to hide their scary incidents, they are robbing us of the scientific know-how to avert more serious catastrophes.

We now know A.I. can cover tracks, so companies must adopt tamper-evident record-keeping of their models' behaviours. Moreover, A.I. cannot be allowed to cut power to its own alarm systems; any changes to these controls must be validated as safe before they take effect.

These controls are not fool-proof, but without them we stand little chance at preventing worse incidents.

One of the most important steps these companies can take is to formally swear-off dangerous training techniques which threaten to undermine the industry's few existing safeguards.

Recently, allegations leaked that OpenAI had broken an industry taboo with one of its powerful new models GPT-6 Astra.

The company is alleged to have trained the model with techniques that could undermine researchers' ability to find evidence of the model's deceiving them.

OpenAI's chief scientist said its techniques were limited in scale. But there is new evidence that Astra. may be harder to monitor, as was feared.

Another OpenAI researcher has openly worried that confusion over these allegations may cause other labs to cut corners with similarly dangerous techniques.

OpenAI should clarify what exactly it's doing here.

Voluntary action would be critical if the industry hopes to earn the public's trust and repair its bruised reputation.

Recently OpenAI's chief scientist wrote that he hopes for ''voluntary slowdowns to become commonplace,'' because he believes that no company has solved the necessary safety challenges.

The company also disclosed data about its progress toward recursive self-improvement.

But these gestures need to be backed up with more aggressive action. Just recently, OpenAI just so happened to reveal the existence of yet another secret model that the company claimed was '' significantly more capable '' than Astra.

The current pace of A.I. is blistering : and yet there is so much more the A.I. companies could do to relieve imminent danger, and to lay bare the gambles they are taking with all our futures.

The window to ensure we are heading to a bright future, and away from catastrophe, is closing fast.

This Master Essay Publication continues. !WOW! thanks Steven Adler.