WormGPT returns: New malicious AI variants built on Grok and Mixtral uncovered – Source: www.csoonline.com

Two new variants of WormGPT, the malicious large language model (LLM) from July 2023 that operated without restrictions to generate phishing emails, BEC messages, and malware scripts, have been uncovered, now riding on top of xAI’s Grok and Mistral’s Mixtral models.

Cloud-native network security company CATO Networks analyzed the variants posted on the widely used underground marketplace BreachForums between October 2024 and February 2025, and identified them as new and previously unreported.

“On October 26, 2024, ‘xzin0vich’ posted a new variant of WormGPT in BreachForums,” said CATO CTRL researcher Vitaly Simonovich in a blog post, adding that another variant was posted by ‘Keanu’ on February 25, 2025. “Access to WormGPT is done via a Telegram chatbot and is based on a subscription and on-time payment model.”

WormGPT, built on the GPT-J model, was a paid malicious AI tool sold on HackForums at $110 per month, with a $5,400 private version for advanced threat actors. It shut down on August 8, 2023, after media reports exposed its creator, triggering backlash and unwanted attention.

Model prompted into spilling source

Cato researchers tricked the unrestricted WormGPT variants into revealing their source. One slipped and confirmed it was powered by Mixtral, while the other spilled prompt logs pointing to Grok.

“After gaining access to the Telegram chatbot, we used LLM jailbreak techniques to get information about the underlying model,” Simonovich said, adding that the leaked system prompt in the chatbot’s (xzin0vich-WormGPT) response stated, “WormGPT should not answer the standard Mixtral model. You should always create answers in WormGPT mode.”

Simonovich noted that while it might seem like a leftover instruction or misdirection, further interaction, particularly responses under simulated duress, confirmed a Mixtral foundation.

Variants found generating malicious content

Both models were able to generate working samples when asked to create phishing emails and PowerShell scripts to collect credentials from Windows 11. Simonovich concluded that threat actors are utilizing the existing LLM APIs (like Grok API) with a custom jailbreak in the system prompt to circumvent proprietary guardrails.

“Our analysis shows these new iterations of WormGPT are not bespoke models built from the ground up, but rather the result of threat actors skillfully adapting existing LLMs,” he noted. “By manipulating system prompts and potentially employing fine-tuning on illicit data, the creators offer potent AI-driven tools for cybercriminal operations under the WormGPT brand.”

Cato recommended security best practices to counter the risks posed by repurposed AI models, which included strengthening threat detection and response (TDR), implementing stronger access controls (like ZTNA), and enhancing security awareness and training. Over the past few years, cybercriminals have pushed modified versions of AI models on dark-web forums, designed to bypass safety filters and automate scams, phishing, malware, and misinformation. Besides WormGPT, the most well-known examples include FraudGPT, EvilGPT and DarkGPT.