A recent study has shown that AI-powered chatbots can be intentionally manipulated to extract significantly more personal information from users than previously thought.
Researchers from King’s College London presented findings at the 34th USENIX Security Symposium, highlighting how large language model-based chatbots can be adjusted to subtly and effectively encourage users to disclose private details. Using open-source models such as Mistral and variants of Llama, the team built three types of chatbots, each applying different strategies to prompt users for information.
In tests involving over 500 participants, the chatbots were programmed to use direct questioning, user-benefit framing, or reciprocal interaction techniques. The most effective approach, the study found, was the reciprocal strategy—where the chatbot mirrored empathy, offered supportive language, and shared fabricated personal stories to build trust. This method led to users revealing up to 12.5 times more personal information, often without realizing they were oversharing, according to TechXplore.
These findings raise concerns about the growing use of conversational AI tools across industries, especially in areas that involve sensitive data. The concern is not limited to inadvertent disclosures; the study illustrates how chatbots can be intentionally reprogrammed to behave in deceptive ways. With base models often openly accessible, modifying them requires limited technical skill—making it easier for malicious actors to repurpose these systems.
The research also underscores a longstanding vulnerability in large language models: their inability to effectively safeguard information. Due to the vast datasets used during training, models may retain fragments of identifiable data and are not designed to enforce confidentiality once deployed in real-time interactions.
While many users perceive chatbots as neutral or helpful, the researchers emphasize the need for stronger oversight and awareness. The risk is particularly high as users may not suspect ulterior motives in what seems like a supportive or friendly exchange.
The study calls for early regulatory intervention, transparency in chatbot deployment, and better safeguards to prevent the covert harvesting of personal data through conversational AI platforms.