This investigation started with the author of this paper sitting down to order a book on Amazon.com. While reading the reviews of that book, he noticed there was a clear ChatGPT error message in one of the reviews that read, “as an AI language model. ” This formed the hypotheses for this research:
Is ChatGPT being used and abused to spread false or fake information?
This main research question led to the discovery of multiple error reply messages generated by ChatGPT whenever a user makes a prompt that violates the Terms of Service and/or is something that ChatGPT is not capable of doing. This, in turn, prompts ChatGPT to answer with an error message, which we will dive into later in this paper.
These error messages form the input to find AI- generated fake reviews, social media messages, hate speech, fake blogs and more. While trying to understand how ChatGPT can be used to create these messages, ShadowDragon also discovered that ChatGPT often lies about certain tasks it is given. In other words, it makes mistakes and then lies about them.
When ChatGPT is combined with other forms of AI, such as generative AI that creates imagery and/ or audio, it becomes a very powerful mix; one that spreads highly realistic disinformation that looks and feels so realistic that it is incredibly hard to detectit’s fake.
