Company decides to implement ChatGPT for client work until one employee discovers dozens of unchecked AI mistakes: ‘[Now] staff members have to complete mandatory retraining’

What happens when AI confidence meets human oversight?

AI is one of those things that has come into our lives and is not going away. It has come to stay and is going to affect and change the way we live. We’re still discovering how it works and what we can do to get the best out of it, without damaging our work. So it’s basically trial and error until we find the sweet spot of it. 

In the story we have below, a company decided to implement ChatGPT into their normal workload to make it faster, and what they thought it could be more accurate. They used to send clients emails with very delicate financial information, and they really thought they were doing the right thing. However, one employee checked an information that was sent wrongly to a client and caused a big issue.

They discovered that more than 20 emails had been sent without any fact-checking process, revealing and explaining procedures and information of the company that were incorrect. As a consequence, management decided to stop the use of AI and decided to have staff members complete a mandatory retraining, as well as a professional conduct course.

On the other hand, the employee was asked to write some accounting references for his coworkers because he’s the only one who has the accounting degree and is more in touch with the subject. 

Needless to say, IA is not a tool that we can truly trust. It might accelerate some procedures and be helpful for some things, but mainly, it is not really ready to work on its own, so people have to be very careful when deciding to use it. This company now has to apologize to its clients and take back what they said… imagine what that is going to look like.