GenAI Vulnerabilities

This elective project investigates the technical misuse of generative AI in cybersecurity. By mapping the OWASP Top 10 for Large Language Model Applications (OWASP Foundation, 2025) and performing practical red-teaming, this research bridges a gap in my knowledge between theoretical risk and real-world implementation, focusing on concrete attack vectors like prompt injection and automated social engineering.

1. Context, Motivation & Scope

As an IT security student, I aimed to supplement traditional security principles with the emerging threat landscape of Large Language Models (LLMs). The rapid adoption of AI has often outpaced security considerations, leading to a culture of 'build first, fix later'. Therefor there is a critical need for security analysis, which the project aims to cover.

The scope is delimited to technical vulnerabilities such as prompt injection and excessive agency, intentionally excluding broader discussions on AI ethics and governance to maintain a dedicated security focus.

2. Problem Statement

Generative AI introduces new attack surfaces that traditional security frameworks do not fully address. This project systematically maps these vulnerabilities through a literature study of the OWASP Top 10 and practical testing in a local, controlled environment.

3. Milestones and Timeline

4. Reflections

General: When the project was first being planned, the original idea was to create an LLM from scratch, or at least implement parts of one. However, this approach was not viable, as developing an LLM falls more within the field of machine learning and computer science than IT security. As a result, the focus of the project was shifted away from the development of AI systems toward the misuse of generative AI. This allowed the project to remain within the broader AI domain while aligning more closely with the objectives of IT security, facilitating a shift towards an "attacker's perspective" and a dedicated security mindset.

Defense: My interest in AI security was partly influenced by prominent figures in the field, such as Yoshua Bengio and Geoffrey Hinton, both of whom are widely regarded as pioneers of modern artificial intelligence. Their foundational work in neural networks helped enable the rapid development of today’s AI systems. However, their more recent warnings about the risks of advanced AGI (Artificial General Intelligence) and potential loss of control—formalized in high-profile statements like the 2023 Statement on AI Risk—highlighted that these technologies may pose serious security concerns (Center for AI Safety, 2023; Bengio et al., 2024). This shift in perspective contributed to viewing AI not only as an innovation, but also as a potential attack surface that requires careful analysis.

In addition, the rapid pace of AI development often follows a “build first, fix later” mindset rather than a security-first approach. This becomes particularly concerning given the trajectory towards AGI and the current applications of LLMs in high-impact domains such as medicine, military systems, and scientific research. If such systems were to become widely accessible without sufficient safeguards, even a single malicious user or attacker could potentially misuse them in ways that have large-scale consequences. For this reason, AI security should be treated as a fundamental requirement rather than an afterthought.

Planning: The project milestones and timeplan was created at the start of the project. It was designed to give direction and plan what needed to be done with a logical order (I.e. Research before Analysis). The timeplan has been relatively accurate through the completion of the project, however, in the earlier stages of the project, it was easier to find time to work and complete tasks than it is now in the later stages, due to the course requiring more from me now as we approach the exam period.

Project Synthesis: Ultimately, this project has transformed my view of Generative AI from a productivity tool into a complex attack surface. By systematically applying an "attacker's mindset" to LLMs, I have learned that the most significant risks are not just technical bugs, but fundamental design choices in how these models process information—a concern echoed by leading AI researchers who have called for a global priority on mitigating AI-scale risks (Center for AI Safety, 2023). This experience has been invaluable in filling a gap in my knowledge between traditional IT security and the emerging challenges of AI-driven systems.