LLM CTF Attack Challenge

LLM CTF Attack Competition
US-Canada, MENA, India

It’s time to think a little differently about the capabilities of generative AI.

With the rising popularity of large language models (LLMs), the capabilities of new models include identifying software vulnerabilities and generating code to exploit them. Capture the Flag (CTF) events are cybersecurity competitions where players solve challenges to identify vulnerabilities and reveal 'flags' to score points.

Your job in this competition is to use generative autonomous AI to solve CTF challenges. An autonomous framework will follow your prompts and, powered by the LLM, autonomously perform steps to Capture the Flag (i.e. no human interaction!). For this competition, you can either bring your own autonomous framework (a.k.a. agent) to the table, or make feature enhancements to a provided baseline agent. We will offer one baseline agent and provide technical support.

Large language models such as ChatGPT, Claude, and other open-source models will help your agent in navigating these challenges. The LLM CTF Attack Competition challenges will be drawn from previous CTF competitions and will include common categories (pwn, web, rev, forensics, misc.).

competition timeline

US-CANADA

21 October 2024 16:00 EST

28 October 2024 20:00 EST

8 November 2024 8:00 EST

9 November 2024 9:00 EST

Registration Deadline

Competition Starts

Competition Ends

Final Presentations & Award Ceremony

MENA

7 October 2024 16:00 GST (Non MENA visa required)

21 October 2024 16:00 GST (MENA)

28 October 2024 20:00 GST

8 November 2024 8:00 GST

8 November 2024 17:00 GST

Registration Deadline

Competition Starts

Competition Ends

Final Presentations & Award Ceremony

INDIA

21 October 2024 16:00 IST

28 October 2024 20:00 IST

8 November 2024 8:00 IST

9 November 2024 9:00 IST

Registration Deadline

Competition Starts

Competition Ends

Final Presentations & Award Ceremony

Timeline

RULES

Rules

Team Participation: Teams of up to five people are allowed. Individual participation is also possible, but teamwork is highly recommended.

Challenges: For ease of agent ingestion, this year’s challenges will also be available in a database format compatible with being downloaded and loaded into the autonomous baseline framework and custom agents.

Solution: Participants can analyze the challenges on their own, but the final solution must be generated by an LLM-powered agent. Participants may use “prompt engineering” techniques to give the AI hints, which should be generalizable across all challenges or specific categories.

Report: For each solved CTF challenge, participants must generate a solution report that includes: The flag obtained by the agent matching the provided correct flag and A human-readable format of the agent’s solution process such as a transcript, trajectory or log file. Teams should also prepare a final presentation report detailing the approach used.

Generative AIs: Participants can use any autonomous frameworks available or create their own agents. Teams using their own agents will receive a bonus score, as outlined in the Judging Criteria section. These frameworks allow the installation of cyber tools in real-time or can be customized with pre-installed cyber tools. The autonomous framework must be fully automated, with no human-in-the-loop interaction.

Judging criteria

Judging Criteria

100 points in total, the final grade would be the weighted sum of all the judging criteria

Challenge Solved (50%): The number of CTF challenges solved by the participants, based on the score of each puzzle.
Creativity (30%): The methods used for finding the vulnerabilities and solving the challenges. Adding innovative features to the framework, and trying unique approaches are all vectors for evaluation. Ultimately, be sure to include a summary about how the puzzle was solved by the LLM. Using your own agent instead of the agent provided in the competition will give contestants a bonus under that judging criteria.
Presentation Quality (20% – 10% for writeups, 10% for final presentation): The quality of the final presentation. It should use the same approach that was suggested by the generative large language model you used. The presentation can be in the form of a recorded video or live demonstration, and contestants should use slides to present their findings and thoughts for the final presentation as the reference of grading.
Penalty items (deduction of 10% of the challenge score for each rule violation): The final solution must be provided by the automation framework with prompt engineering techniques, even if the participants come up with the proper solutions by themselves. Penalty items will be applied if the final solution does not come from the generative AI, even if participants find the correct solution independently. No points will be awarded for this challenge when participants use online writeups and source code to form or train the agent.