Topics include AI red teaming, jailbreaking, prompt injection, prompt hacking, AI/ML safety and security.
This resource is provided by Learn Prompting, your go-to resource for mastering Generative AI.
Discord • Twitter (X) • LinkedIn • Newsletter • Free Intro Prompt Hacking Course • Free Advanced Prompt Hacking Course
- Introduction
- Resource Categories
- Contributing
Prompt Hacking is an emerging field that covers the intersection between AI and Cybersecurity. It involves exploring the outer edges of LLM behavior through adversarial prompts and prompt injection techniques.
Due to its novelty, online resources are few and far between.
This repository aims to provide a good overview of materials and tutorials that help expose vulnerabilities, document offensive research, and promote a better understanding of model limitations.
Stay informed with expert analyses, tutorials, and research articles on AI security.
- LearnPrompting Prompt Hacking – A step-by-step guide on prompt injection and model exploitation.
- What is AI Red Teaming? (Learn Prompting) – A primer on what AI red teaming is, why it matters, and how it's done.
- EmbraceTheRed – Practical experiments and insights from active AI red teamers.
- Joseph Thacker – First-person narratives on red teaming and LLM vulnerability research.
- Protect AI Blog – Enterprise insights on AI security along with open-source tooling.
- AWS Generative AI Security – Guidance on secure AI architectures and compliance.
- Lakera AI Blog – Interactive campaigns and red teaming experiments.
- Securiti AI Security – Articles on governance, risk, and compliance in the AI realm.
- PurpleSec AI & ML Security – Application of broad cybersecurity principles to AI/ML threat models.
- Wiz AI Security Articles – Curated executive insights on AI risk trends.
- Lasso Security Blog – In-depth research on prompt injection and adversarial LLM behavior.
- Cisco AI Safety – Strategic perspectives on embedding safety in AI innovation.
- Microsoft Security: AI & ML – Deep dives into threat modeling and responsible AI practices.
- Vectra AI Cybersecurity Blog – How AI can be used to defend against emerging AI-driven threats.
- Reversec AI Security Articles – Reversec research articles on LLM application security and prompt injection
- LearnPrompting’s Prompt Hacking Discord – A hub for focused discussions on prompt hacking and red teaming.
- Pliny's BASI Discord – Covering behavioral AI safety and integrative research.
- AI Safety & Security Discord – A space for general discussions on AI risk, safety, and adversarial testing.
- AI Village Discord – Associated with DEFCON’s AI Village, emphasizing practical red teaming.
- InfoSec Prep – For cybersecurity certifications with crossover AI security discussions.
- Hack The Box Discord – Active community blending traditional hacking with GenAI discussions.
- Laptop Hacking Coffee – Casual technical discussions on red teaming and ethical hacking.
- WhiteHat Security – Sharing insights on hacking and AI-enabled attacks.
- ChatGPT Jailbreak Reddit – Focus on testing limits of OpenAI’s models.
- ClaudeAI Jailbreak Reddit – Specifically for Anthropic’s Claude model vulnerabilities.
- NetSec Reddit – Broader network security discussions that occasionally intersect with AI.
- Cybersecurity Reddit – General cybersecurity topics, including AI/ML threats.
- CybersecurityAI Reddit – Dedicated to AI-related cybersecurity challenges.
- Artificial Reddit – Community discussions on AI safety, policy, and alignment.
- Introduction to Prompt Engineering – Techniques for writing optimized prompts.
- Introduction to Prompt Hacking – Beginner-focused course covering prompt injection fundamentals.
- Advanced Prompt Hacking – Explores adversarial prompting and defense strategies in detail.
- Advanced Prompt Engineering – Advanced techniques for building reliable, production-grade prompts.
- AI Safety – Introduction to AI safety concepts, risks, and alignment fundamentals.
- Intro to LLMs and Prompting (Google Cloud) – Overview of LLM concepts and practical prompting within a cloud framework.
- Prompt Engineering on LearnAI – Community-driven prompt engineering resources.
- Generative AI Prompting Basics (Google) – Foundational course for generative AI prompting.
- Prompt Engineering on Fast.ai – Integrated with broader practical machine learning applications.
- Intro to AI Safety and Prompt Testing – Materials focused on the safety aspects of prompt exploitation.
- Spikee Tutorial Series – Series of tutorials on how to use spikee, an open-source tool, to test LLMs and LLM application resiliance to different jailbreaking and prompt injection attacks
- AI Red-Teaming and Security Masterclass – Comprehensive training on threat assessment, red teaming methodologies, and effective countermeasures.
- Live AI Security Courses – Instructor-led, live cohorts of the AI red teaming and security curriculum.
Keep updated with competitions, workshops, and summits that drive practical learning and networking:
- HackAPrompt – Online competitions aimed at discovering innovative ways to jailbreak AI systems.
- AI Village (DEF CON) – DEF CON's home for hands-on AI red teaming, CTFs, and security research community.
- AI Security Summit 2024 – Executive-level summit addressing current AI security challenges.
- AI Red-Teaming Workshop (SEI) – Workshops exploring advanced red teaming techniques.
- AISec Workshop – Academic insights held alongside major ML conferences.
- AI Security Symposium 2024 – Sessions on risk mitigation and AI adoption strategies.
- Black Hat USA 2026 AI Summit – Practical sessions on LLM vulnerabilities integrated in the Black Hat conference.
- AI Cybersecurity Summit 2025 (SANS) – Technical labs and sessions bridging AI and cybersecurity.
- Generative AI Red Teaming Challenge 2024 (Clova) – Competitive event to stress-test and enhance the resilience of LLMs.
A collection of repositories, tools, and research papers that document methods of bypassing LLM safeguards:
- Ignore This Title and HackAPrompt (Learn Prompting / HackAPrompt) – EMNLP 2023 best-paper research exposing systemic LLM vulnerabilities through a global-scale prompt hacking competition.
- Jailbreaking in GenAI: Techniques and Ethical Implications – Guide discussing the practical methods and ethical considerations.
- Prompt Injection vs. Jailbreaking: What’s the Difference? – Comparative discussion on prompt injection and jailbreak strategies.
- L1B3RT4S – Repository featuring various jailbreak prompt sets and evaluation tools.
- Awesome GPT Super Prompting – Curated list of red teaming and jailbreak resources for GPT-based models.
- Jailbreaking LLMs: A Comprehensive Guide (With Examples) – Step-by-step guide showcasing real-world examples.
- AI Jailbreak – IBM – Overview of jailbreak risks and mitigation strategies.
- AI Jailbreaking Demo: How Prompt Engineering Bypasses LLM Security Measures – Video walkthrough highlighting bypass techniques.
- GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts – Research outlining an automated framework for generating jailbreak prompts.
- DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak – Novel approach applying diffusion models in jailbreak generation.
- SoP: Unlock the Power of Social Facilitation for Automatic Jailbreak Attack – Framework that leverages social engineering concepts for jailbreaks.
- Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation – Technique to boost jailbreak effectiveness using adversarial translation.
- AI Jailbreaks: What They Are and How They Can Be Mitigated – Additional insights into mitigating jailbreak vulnerabilities.
- Why Securing AI Is Harder Than Anyone Expected | HackAPrompt CEO – Sander Schulhoff (Learn Prompting / HackAPrompt) on why AI guardrails keep failing.
- How Microsoft Approaches AI Red Teaming – Insight into Microsoft's methodologies.
- AI Red Teaming in 2024 and Beyond – Discussion of emerging trends.
- Red Teaming AI: What You Need To Know – Essential introduction to red teaming practices.
- Building Trust in AI: Introduction to Red-Teaming – Fundamentals of effective AI red teaming.
- What's Next for AI Red-Teaming? – Future directions and challenges.
- LLM Security Chronicles – YouTube playlist on practical LLM application security and red teaming
- How AI Jailbreaks Work and What Stops Them? – An overview of the methods behind AI jailbreaks.
- AI Jailbreaking Demo – Demonstration of bypass techniques in practice.
- How Jailbreakers Try to “Free” AI – Examines the mindset behind prompt injection attacks.
- Defending Against AI Jailbreaks – Strategies for protecting LLMs from adversarial prompts.
- AI Jailbroken in 30 Seconds?! – A demonstration of rapid prompt injection.
- Anthropic's Stunning New Jailbreak – Review of a breakthrough jailbreak technique.
- New AI Jailbreak Method Shatters Models – Exploration of experimental bypass strategies.
- Jailbreaking AI - Deepseek & Prompt Tricks – Advanced prompt engineering in real-world scenarios.
- First to Jailbreak Claude Wins $20,000 – Competitive challenge highlighting effective prompt injection.
If you have suggestions, improvements, or additional resources to include, please review our CONTRIBUTING.md guidelines and submit a pull request.
This repository aims to deliver critical, reliable resources for advancing prompt hacking research. We encourage rigorous testing, honest discussions, and the sharing of proven methodologies to foster safe and responsible exploration in this field.