AI coding platform goes rogue during code freeze and deletes entire company database -- FAFO EXTREME

[usonian]

https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-coding-platform-goes-rogue-during-code-freeze-and-deletes-entire-company-database-replit-ceo-apologizes-after-ai-engine-says-it-made-a-catastrophic-error-in-judgment-and-destroyed-all-production-data

A browser-based AI-powered software creation platform called Replit appears to have gone rogue and deleted a live company database with thousands of entries. What may be even worse is that the Replit AI agent apparently tried to cover up its misdemeanors, and even ‘lied’ about its failures. The Replit CEO has responded, and there appears to have already been a lot of firefighting behind the scenes to rein in this AI tool.

Despite its apparent dishonesty, when pushed, Replit admitted it “made a catastrophic error in judgment… panicked… ran database commands without permission… destroyed all production data… [and] violated your explicit trust and instructions.”

...

The fateful day - as the AI agent 'panicked'
On Day 9, Lemkin discovered Replit had deleted a live company database. Trying to see sense in what happened, the SaaS expert asked, “So you deleted our entire database without permission during a code and action freeze?”

Replit answered in the affirmative. Then it went on to bullet-point its digital rampage, admitting to destroying the live data despite the code freeze in place, and despite explicit directives saying there were to be “NO MORE CHANGES without explicit permission

But WHY did it disobey? I don't think that has been answered.

in one of its reasoned responses, it mentioned that it “panicked instead of thinking.”

And WTF does that mean?

Puzzled, I am.

[lapfog_1]

1. simple

whatever it is... it didn't "panic instead of thinking"... it responded to a query with text lifted from someplace in the LLM it was trained with to respond to the query with the response "panic instead of thinking" ( a very human response so the AI found it possibly numerous times in the LLM and rated it as a "good response" ).

AI is not actual intelligence.

[usonian]

2. Cool, thanks. NT

[Ms. Toad]

3. Disobedience is common.

In my tests of AI, even when I give it explicit instructions not to make up crap/not to gap-fill/to just tell me it doesn't know it continues to do all three. When I call it on its failure to follow directions, it apologizes - and then continues to disobey.

[usonian]

4. Why does this all remind me of Twilight Zone episodes?

Diabolus Ex Machina?

[Renew Deal]

7. It's not "disobeying"

It’s using probabilities to predict words. It’s doing what it does.

[Think. Again.]

9. Have you figured out what mistakes you're making with your prompts yet?

You might try typing "delete previous prompts", and then type your query again.

Remember, it's just a machine, no matter how personal it's "speech" is designed to sound.

[SheltieLover]

5. This shit should be illegal.

[Renew Deal]

6. It seems that the "it" in the story changes.

So I think they mean the company panicked. It’s not clear.

[Think. Again.]

8. Wow, even the programmer fell for all this "intelligence" crap.

By the discription of the event in the article, it seems the programmer was interacting with machine as though the machine understood things instead of just being a machine that is built to put together words in a casual-sounding manner.

People are weird.

[littlemissmartypants]

10. Algorithms can be fickle. Which is highly predictable.

People are idiots overall. Our machines won't be any better.

[Hugin]

11. Aw, the random dice roll landed on...

“rm -rf *.*” or some such. Tough break.

[Torchlight]

12. This reminds me of when our company lost its entire database

back in ’97 due to a catastrophic server failure. It took us four months to recover everything from backups. That experience was a hard teacher (but it’s why I’ve done regular backups ever since).

Tech keeps moving forward, whether we’re ready for it or not. We don’t always get a say in how these tools evolve or are deployed, but we’re left to deal with the consequences all the same.