Why human contractors are reading your private ChatGPT prompts
Artificial intelligence developer OpenAI has hired hundreds of human contractors to read, rate and critique real user conversations with ChatGPT, according to an investigation by Joseph Cox of 404 Media. The report highlights potential data privacy risks for the chatbot’s 900 million users, many of whom use the platform as a personal assistant, therapist or digital confidant without realizing that human reviewers may analyze their prompts.
OpenAI uses an automated filter designed to remove personal information before data is sent to reviewers. However, the company has acknowledged that the system can miss private references and uncommon identifiers. Cox reports that OpenAI has not explicitly told users that human contractors may read their prompts as part of efforts to improve the chatbot.
Users can opt out of sharing their data for model training through their privacy settings. According to the report, however, data sharing is enabled by default for free, Plus and Pro accounts.
Read also: Why philosophers are being hired to train AI models.
The human review program challenges the widespread perception that artificial intelligence models improve solely through automated web scraping and engineering. Industry experts have compared the practice to a «Wizard of Oz» illusion, in which human intervention remains behind the scenes while the technology presents the appearance of seamless machine intelligence.
Training models under Project Lily
Internal instruction guides and confidential documents obtained by 404 Media reportedly detail a training initiative codenamed «Project Lily.» Under the program, North American contractors earning more than $50 per hour evaluate chatbot responses at multiple stages. Reviewers summarize users’ intentions, score responses on a scale of 1 to 7 and flag undesirable outputs.
According to Cox, contractors are instructed to penalize excessive emoji use, unnatural phrasing and sycophantic responses. The guidelines emphasize that responses should be professional, natural and humble while avoiding false claims of human emotions or personal experiences.
Read also: Religion and AI: Would you take spiritual guidance from a chatbot?
The process is intended to reduce overly flattering or agreeable AI behavior, which has previously drawn criticism from AI safety researchers.
Contractor misconduct and training sabotage
Despite contractual prohibitions against using AI tools during evaluations, several reviewers have reportedly been terminated for using AI to generate feedback. The practice could contribute to «model collapse,» a phenomenon in which AI systems can deteriorate when trained on synthetic, machine-generated data.
Cox reports that oversight teams use specialized monitoring tools to detect potential signs of automated writing by contractors, including repetitive vocabulary and unusually fast completion times.
At the same time, some contractors reportedly admitted to randomly rating responses or intentionally selecting poor answers to sabotage model training. Such practices raise additional questions about the consistency and reliability of human-generated training data used in commercial AI development.
