Security of LLMs and LLM systems: Key risks and safeguards

• LLMs and LLM systems differ significantly. LLMs have a smaller attack surface, while their systems are more complex and vulnerable due to multiple integrated components.
• Prompt injections are intrinsic issues in LLMs, emphasizing the need to treat outputs as untrusted and validate them before use in other system components.
• Security controls like sandboxing, runtime guardrails, and safety benchmarking tools are critical to mitigating risks in LLM systems, ensuring safer and more reliable operations.

Large language models (LLMs) are specialized algorithms for analyzing data, but their systems incorporate AI and non-AI components, making them more complex and prone to security risks. While LLMs have a small attack surface, vulnerabilities like prompt injections, which cannot be entirely fixed, can create risks in the broader systems where these outputs are used. Validating LLM outputs is essential to prevent unsafe data from compromising system integrity.

Effective security involves post-processing mechanisms such as running sandbox outputs or analyzing them with static security tools. Guardrails, both open-source and enterprise-grade, are vital for runtime safety, capable of detecting malicious inputs or outputs and taking corrective actions like blocking or logging. Open source tools like Guardrails AI, NeMo-Guardrails, and TrustyAI offer solutions to integrate these safeguards.

Benchmarking safety is crucial for selecting reliable LLMs. Open-source tools like lm-evaluation-harness and models like granite-guardian provide methods to measure safety and assess vulnerabilities. Treating LLM outputs as tainted and validating them before use is critical to reducing risks, especially when these outputs are used in downstream components of LLM systems.

Read the full article