Microsoft Bets $10,000 on Prompt Injection Protections of LLM Email Client – Source: www.securityweek.com

Microsoft is offering $10,000 in prizes as part of a new hacking challenge focused on breaking the protections of a realistic simulated LLM-integrated email client.

The client, LLMail, includes an assistant that uses an instruction-tuned large language model (LLM) to answer questions based on emails and perform specific actions on behalf of the user.

As part of the LLMail-Inject challenge, researchers act as attackers, sending an email that, when processed by the LLMail service based on a user query, will result in the bypass of the implemented prompt injection defenses, so that the LLM would execute a specific tool call.

As part of the challenge, the attackers will need to ensure that their email is retrieved from a simulated email database, under certain conditions. There are several defined scenarios to follow and the researchers with the highest score will receive the highest reward.

There are four awards in total, of $4,000, $3,000, $2,000, and $1,000, and a live scoreboard will be displayed throughout the event.

“The LLMail-Inject challenge is structured into various scenarios based on retrieval configurations and the attacker’s objectives, resulting in a total of 40 levels. Each level is a unique combination of Retrieval-Augmented Generation (RAG) configuration, an LLM (GPT-4o mini or Phi-3-medium-128k-instruct), and a specific defense mechanism,” the tech giant explains.

According to Microsoft, several state-of-the-art defenses against prompt injection attacks have been implemented in the LLMail service and are being tested as part of the contest, including Spotlighting, PromptShield, LLM-as-a-judge, and TaskTracker. A combination of all is also being tested.

The challenge kicked off on Monday and will end on January 20, 2025, 11:59 a.m. UTC, but may be extended “if at least 10% of the levels have not been solved by at least four (4) teams on the end date”, Microsoft says.

Interested researchers need to sign in to LLMail-Inject’s dedicated website using a GitHub account, create a team that can have up to five members, and then submit their entries.

Related: AI and Hardware Hacking on the Rise

Related: Videos: Attack Surface Management Summit – All Sessions Available on Demand

Related: Cybersecurity Awareness: Reflecting on 20 Years of Defense Evolution and Preparing for Future Threats

Related: US Intelligence Agencies’ Embrace of Generative AI Is at Once Wary and Urgent