Half of Security Pros Want GenAI Deployment Pause – Source: www.infosecurity-magazine.com

Around half (48%) of security professionals believe a “strategic pause” in generative AI deployment is needed to recalibrate defenses, according to a new report by offensive security firm Cobalt.

Most (94%) of security leaders and practitioners surveyed said they have observed a significant increase in the adoption of genAI within their industry over the past 12 months.

Worryingly, 36% of respondents admitted that the rate of genAI deployment is moving faster than their teams are able to manage.

Despite many security professionals expressing a desire for a strategic pause in genAI deployment, Gunter Ollmann, CTO, Cobalt, warned that this isn’t realistic given the benefits this technology provides to businesses.

“Threat actors aren’t waiting around, and neither can security teams. Our research shows that while genAI is reshaping how we work, it’s also rewriting the rules of risk. The foundations of security must evolve in parallel, or we risk building tomorrow’s innovation on today’s outdated safeguards,” he commented.

Top GenAI Security Vulnerabilities

Around three-quarters (72%) of security practitioners cited genAI as their top IT risk.

The top AI risks cited by respondents related to AI data security and accuracy. Sensitive information disclosure was the most commonly cited risk (46%), followed by data model poisoning and theft (42%), inaccurate data (40%) and training data leakage (37%).

Despite the high level of concern, 33% of security teams are failing to conduct regular security assessments, including penetration testing, for their large language model (LLM) deployments.

Around a third (32%) of all vulnerabilities discovered in genAI tools are classified as serious (high or critical risk), according to assessments undertaken by Cobalt since 2022, when LLM testing began.

This is the highest proportion of serious vulnerabilities found across all asset types, according to the researchers.

Just 21% of these serious vulnerabilities are actually resolved, the lowest resolution rate among all types of penetration tests conducted.

SQL injection, a classic web application vulnerability, was the most common vulnerability discovered in these genAI assessments, at 19.4%. Another traditional web vulnerability, stored cross-site scripting, also ranked high at 9.7%.

Cobalt said that these findings demonstrate that foundational web application security best practices remain highly relevant when deploying LLM applications. These include robust input validation, secure coding practices and proper configuration of underlying systems.

A number of LLM-specific vulnerabilities discovered in pentests were also highlighted in the report. These included:

• Prompt injection leading to inappropriate content: Pentesters discovered that by carefully crafting input prompts, they could bypass the LLM’s content filters and instructions, allowing them to elicit inappropriate responses
• Prompt injection leading to sensitive data exposure: Prompt injection techniques were able to trick the LLM into revealing sensitive data, such as personally identifiable information from the dataset
• Model denial of service: By sending a high volume of complex or resource-intensive queries to the LLM endpoint via its API, the pentesters could overwhelm the model, resulting in extreme slowdowns or complete service unavailability
• Excessive agency: Pentesters were able to make the LLM perform actions and access information beyond its intend scope through a series of nuanced interactions and prompt manipulations

“These case studies highlight that uncovering many LLM-specific vulnerabilities, especially those involving nuanced prompt manipulations or complex interaction chains, requires significant human expertise and creative, adaptive testing approaches,” the report read.