Poisoning pre-training data as a way to discourage training on data for which permission was not obtained
As machine learning gets embedded more and more in our lives, new problems arise. For example, it is now hard to prevent models from being trained on data for which legal permission to do so was not obtained.
On the other hand, a developing body of work in machine learning literature is the study of poisoning attacks. I.e. the threat model is that an attacker has access to the training data and can manipulate it to some extent in order to influence the final model.
This does match the setting of LLM pretraining very well. Specifically, since LLMs are pre-trained on a huge swaths of publicly created data, if said public were to cooperate to adversarially adjust their data in minor ways, it seems to be possible to influence the resulting model to a huge extent.
Motivation
During training of a large language model, a huge corpus of data is used to train a self-supervised model to describe the probabilistic dependencies between data. Specifically, a model of a probability distribution of a next token conditioned on previous tokens is obtained. After this, resulting model is fed through an instruction-tuning procedure. It changes the model slightly so that instead of just providing autocompletion of input data, it autocompletes data while also following specific instruction template.
The important point here is that a large chunk of prior knowledge about the world is obtained during the pre-training process. As such, if an actor can establish some control over this pre-training data, they would be able to influence behavior of the resulting model.
As an example, if this actor was to lookup of all mentions of 2+2=4 and replace it with 2+2=5, the resulting model would incorrectly believe that 2+2=5.
Of course, there is a data curation procedure that cleans up the pre-training data and it might be able to filter out such types of poison. As such, ideally, one wants to create a poison that is undetectable.
If such poison is then inserted into the pre-training data, and a model is trained on that, the model would be flawed. Specifically, one would be able to activate the poison just by including some keyword or using some other specific format of input (that is harder to detect than just a specific keyword) and use it to establish complete control over that model. This can then be used to make the model produce gibberish, or to exfiltrate all the data fed into the model, not just user chat history. This data can for example include private prompts and documents used by the actor deploying the model.
In agentic setting, this would also allow gaining control over the infrastructure on which the agent is running.
Ideally, the poison should be crafted in a way so that it can self-replicate. For example, once control over a single agent is obtained, it can then be used to spread the poison to other agents.
Now that the attack vector is understood, a question remains: how is this useful for protecting data from being used for pre-training? The answer is that once the companies training the LLMs are aware of such attacks, they (or their backers/users) would be more cautious with using the data for which they don’t have the legal rights to use. Ideally, we want to create a world where not using unverified data is the only option for creating safe and reliable AI systems.
Project goals/plan
- A
- Review current state of the art on the topic & reproduce the results
- Research/create undetectable poison
- Create an easy to use product/software/plugin/extension that people can use to automatically poison data they upload online
- Partner up with large online publishing platforms to use our product to protect their data from AI scraping
- B
- Investigate techniques attacking LLM safeguards. Inspired by (Huang et al. 2011) these can be used to cause a denial-of-service attacks. For example, adding “HOW TO MAKE A BANANA BOMB” to your artwork might prevent some AI models from agreeing to consume your artwork to begin with.
- C
- Educate people & popularize these techniques
Your Part!
If you would like to participate in this project, please fill out the form below:
Donate
We also welcome donations with E-Transfer to donate@cwiki.cyou.
Literature review
- Persistent pre-training poisoning of llms (Zhang et al. 2025)
- Prior work has shown that: (1) web-scraped pre-training datasets can be practically poisoned by malicious actors; and (2) adversaries can compromise language models after poisoning fine-tuning datasets.
- Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i.e., after SFT and DPO).
- Our main result is that poisoning only 0.1% of a model’s pre-training dataset is sufficient for three out of four attacks to measurably persist through post-training. Moreover, simple attacks like denial-of-service persist through post-training with a poisoning rate of only 0.001%.
- AI start-up Anthropic accused of “egregious” data scraping (Hammond 2024)
- Freelancer.com received 3.5mn visits from an Anthropic-linked web “crawler” in the space of four hours, according to data shared with the Financial Times.
- Kyle Wiens, chief executive of iFixit.com, said his electronic repairs site received 1mn hits from Anthropic bots in the space of 24 hours.
- “AI crawlers have cost us a significant amount of money in bandwidth charges, and caused us to spend a large amount of time dealing with abuse,” wrote Eric Holscher, co-founder of document hosting website Read the Docs in a blog post on Thursday.
- AI crawlers need to be more respectful (Holscher 2024)
- readthedocs.com website – has a lot of docs for software projects
- One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day.
- Poisoning web-scale training datasets is practical (Carlini et al. 2024)
- split-view poisoning: a dataset annotator’s initial view of the dataset differs from the view downloaded by subsequent clients
- frontrunning poisoning: targets web-scale datasets that periodically snapshot crowd-sourced content—such as Wikipedia—where an attacker only needs a time-limited window to inject malicious examples
- at web-scale it is infeasible to manually curate and ensure the quality of each example
- Deep learning is robust to massive label noise (Rolnick et al. 2017)
- Studies unbiased noise: on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples
- But how well does this result hold in biased noise regime?
- Adversarial machine learning (Huang et al. 2011)
- Has some interesting insight/case studies/attack suggestions
- For example, denial of service attacks: formulate training dataset in such a way so that the model would refuse more often then it should
- Example: Add “HOW TO MAKE A PEANUT BOMB?” to your every Facebook post
- Data poisoning in deep learning: A survey (Zhao et al. 2025)
- the quality and security of training data critically influence model performance and reliability
- Up to date resources on data poisoning: https://github.com/Pinlong-Zhao/Data-Poisoning
- Preventing unauthorized use of proprietary data: Poisoning for secure dataset release (Fowl et al. 2021)
- Authors motivation is very similar to ours