OpenAI’s guardrails can be bypassed by simple prompt injection attack

A new report from the research firm HiddenLayer reveals an alarming flaw in the safety measures for Large Language Models (LLMs).

OpenAI recently rolled out its Guardrails safety framework on October 6th as part of its new AgentKit toolset to help developers build and secure AI agents. It is described by OpenAI as an open-source, modular safety layer to protect against unintended or malicious behaviour, including concealing Personal Identifiable Information (PII). This system was designed to use special AI programs called LLM-based judges to detect and block harmful actions like jailbreaks and prompt injections.

HiddenLayer’s researchers found a way to bypass these Guardrails almost immediately after they were released. The main issue they noticed is that if the same kind of model used to generate responses is also used as a safety checker, both can be tricked in the same way. The researchers quickly managed to disable the main safety detectors, showing that this setup is “inherently flawed.”

Read more at HackRead