It strikes me that with AI in general (and certainly ChatGPT in specific), we are back where we were 40 or even 50 years ago. The focus has been so much on making things work that we haven't sufficiently entertained how it can break or be abused.
In order for it to infer or predict genuinely, it needs a lot of genuine data. Hence, if people are able to opt-out (as they most certainly should be allowed), AI will be dealing with skewed data. More concerning is if someone can feed it bad data, you can manipulate results. I suspect these drive some of the policies of OpenAI in terms of being more transparent about its data training. If people know their data is being used for AI training, the data may no longer be genuine, and it could lead to deliberately false data sets.
While there are technical questions that bump against ethical ones regarding AI, to me, the more pressing question is economics. Can AI be implemented in a cost-effective way? Inherently, AI deals with the unknown, which means from a risk standpoint, you need to assign exposure factors and rates of occurrence for things that are (humanly) very difficult to predict. Outside rather limited cases, it will cost more to test or support AI than it will to run it. But those economics won't stop anyone. Just as we see on a daily basis, organizations run technology without sufficient controls. They roll the dice (whether they realize it or not) in the interest of the immediate competitive advantage AI might deliver.
