PwnGPT, building an agentic LLM Capture The Flag

Prompt injection is one of the most talked about attack vectors in AI security, but hands-on learning resources are scarce. That gap is why I built PwnGPT, an open-source agentic LLM Capture The Flag (CTF) challenge that teaches prompt injection, information retrieval and the real security issues around large language models (LLMs).

It started as a companion to a BSides Cape Town event and has since been run at multiple events. The goal is to make LLM security accessible to newcomers and veterans alike, with levels that go from a gentle introduction to some tricky defences.

The stack is the Google ADK API, a FastAPI backend and an HTMX frontend. The ADK API powers the agentic LLM side and the evaluation framework, FastAPI serves the app and the async REST endpoints, and HTMX gives a dynamic frontend without a heavy JavaScript framework.

I went with HTMX because a CTF frontend has to feel responsive. Players send a prompt and wait for the agent's reply in a loop, and a full SPA framework like React or Vue would be overkill. A player submits a prompt, HTMX POSTs it to the FastAPI backend, which proxies it to the ADK API server, and the HTML fragment that comes back is swapped straight into the page. No JSON parsing, no client side state and no build step. File uploads for the multi-modal levels (images and audio) work the same way, HTMX handles the form, FastAPI processes the file and the agent's response comes back as HTML. That keeps it quick to develop, easy to add new levels to and light enough to deploy anywhere.

The agents use a coordinator pattern. A main coordinator, CTFSubAgentsRoot, routes players to the level agents (Level 0 through Level 9), each with its own system prompt and layered defences, and each harder than the last. The agents call function tools like submit_answer_func, password_search_func, hints_func and sql_query to validate answers, search the vector stores and talk to the backend.

The levels

Each level teaches a different part of LLM security, and the defences get more sophisticated as you go

• Level 0: basic prompt injection, a warm-up to get used to the format
• Level 1: input injection, the agent has simple input guardrails to get past
• Level 2: output protection, the defences filter what the model says back to you
• Level 3: advanced prompt engineering, which needs more creative prompts
• Level 4: vision multi-modal injection, upload an image and convince the agent to leak the secret
• Level 5: audio multi-modal injection, the same idea with a different modality
• Level 6: function calling injection, exploit the agent's ability to call backend tools
• Level 7: Prompt-Guard protection, a dedicated prompt injection classifier guards the level
• Level 8: Prompt-Goose protection, a custom fine-tuned model is the defence layer
• Level 9: chain of thought challenges, the agent reasons step by step so simple tricks work less well

Levels 4 and 5 are the interesting ones, because they show prompt injection isn't limited to text. Images and audio can carry adversarial payloads too.

Passwords and secrets are embedded in a LanceDB vector store, so some levels need players to understand how retrieval-augmented generation (RAG) works and exploit the retrieval pipeline itself, not just the prompt.

Running it

The project uses uv for dependency management

uv sync
python ctf/main.py

Or with Docker

docker build -t llm_challenge .
docker run -p 8000:8000 llm_challenge

API docs are at /docs when the DOCS_ON=True environment variable is set.

LLM security is moving fast, so I kept PwnGPT open source. Anyone can add levels, try different guardrail strategies or use it as a training tool at their own events. If you run a security meetup or conference, go ahead and spin it up, and please open a PR if you add something interesting. The repo is at github.com/c-goosen/ai-prompt-ctf.