The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

I see, thanks. Yes, in that case, I don’t see how this can possibly not constitute copyright infringement.

said with confidence, however Silverman et al Judge explicitly rejected what you just asserted AFAIK

I'm playing stupid now: I believe that if I ask the LLM to "display Harry Potter Book 1" and it does, word-by-word, then you're 100% right, it's copyright infringement. But, if I ask the LLM to "give me an analysis of the "Professor Severus Snape's" character and it gives me one, then I don't see the problem.

So in that sense I understand the response that "they don't violate copyright" by studying the material. Again, I don't pretend to be a lawyer, and not every law has to follow my logic.

Summary books for example are legal, so there is some threshold of compression where things are fine.

On top of that, add patent and trademark law which also ban things A.I.'s might generate from training data. In the case of patents, they can't reproduce the invention even if it's an independent creation. From there, the damages go up if they did it on purpose. That some are trained on patent filings and research papers about patented inventions is just asking to be in a patent suit eventually.

And that's legitimate inventions I'm talking about. Just wait until the patent trolls figure out how to get the A.I.'s to divulge patent violations to sue the A.I. suppliers.

Google keeps coming up with new ways to statistically infer training set data from models. So it's not entirely lossless. At the very least, models that have been trained on a particular work are unusually good at compressing[0] those works, relative to other valid text.

In terms of fair use, one of the larger factors is the 'market substitution' factor, which basically means "does this use compete with otherwise licensed uses that people would ordinarily pay for?" AI absolutely does compete with human artists for the same market. In fact, it's winning handily[1], because you don't have to pay human artists. AI art models absolutely shouldn't be trained on anything with copyright on it.

The other factors don't fare much better. Nature of the original work will differ based on the plaintiff, but the purpose and character of the AI's use of that work is very much commercial. And the amount and substantiality of the use is complete and total. I don't see AI being fair use - at least, not in every one of the many, many training lawsuits currently ongoing against OpenAI and Stability.

[0] Starting with any body of text, an LLM, and an empty context window, compute the next-token probabilities and take the highest one. If it matches the source text, output a 1 bit. If it doesn't, output 0 followed by the ID of the correct next token. Add the correct token to the context window and repeat until the text has been fully compressed. This produces a list of perplexities (wrong words) for the given text which can be used to guide the LLM to output the original work.

[1] Hey, remember when both WotC (biggest art commissioner on the planet) and Wacom (hardware vendor that sells art tools and payment terminals[2]) both got caught using AI art after making very loud and public pledges to not do that? They both wound up buying stock photography on marketplaces that are absolutely flooded with AI trash.

[2] All the credit card readers in Japan are built by Wacom, which is really funny as an artist

But saying "this is fair use" is no more incorrect than saying "this is red".

When a court determines that it isn't, you can continue to argue it as much as you like (to deaf ears), and yet you're still liable to the copyright holder. Whether it's "incorrect" or not is then irrelevant. Let's not argue semantics here.

But saying "this is fair use" is no more incorrect than saying "this is red"

No, they are different. "fair use" is a legal term. It is not like saying "I use it like this, I think it is fair!", the term "fair use" literally is a legal term that means a particular thing in the court of law.

https://en.wikipedia.org/wiki/Fair_use

They're hoping that $10 billion will buy them the kinds of lawyers Oracle had when they basically rewrote copyright law on API's. The AI companies hope to do this in a way that makes everything they're doing either legal or too muddy for lawsuits.

I have not! What's your preferred background reading on this?

so just like Office Space? (paraphrasing) "We steal a fraction of a cent from each transaction, who do we hurt? Nobody. We just put the remainder into our account!"

Sorry that's not how damages are calculated in the US tort system.

The penalty is up to $150k per violation.

It will produce "something more" for the already-wealthy who control the technology. For instance, LLMs will eliminate the need for some customer service jobs, increasing the profit margin for the existing executives and shareholders, while eliminating entry-level jobs from the job market.

That sounds nice except that it didn't happen. They scraped this off many sites whose authors published the material in ways that wouldn't legally allow that. They often publish for a mix of self interest and public benefit. The more you look, the more you find that much altruism is actually self-interest at work. Some have specific terms, too. Let's look at examples in The Pile.

It might be something that's part of their job, requires citations since they value credit, optionally bans commercial use, and maybe has a patent. Arxiv papers are a mix of that. Many on StackOverflow and Hacker News want attribution with some asserting copyright in their comments. Film producers usually want the subtitles to accompany sales of their movies. For FreeLaw, the material is public by law about people who might have never even wanted to testify or be remembered. FreeLaw itself was seeking donations for its service with an additional request to protect the names of people in the dataset. StackOverflow's license explicitly bans copying their data without permission with many individual users also wanting self-promotion to happen side by side with their answers.

So, many things that are in The Pile are works people published hoping to gain some benefit in return. They often had terms that banned their reproduction in ways that prevented them from getting that benefit. They were just public for humans to read and learn from. Some allowed sharing but just wanted credit.

The A.I. users of these works ignore all of that by taking what they made conditionally available without meeting the conditions that benefit the authors. Whereas, if you got the same content on Hacker News or Arxiv, the authors might benefit from it. Even a pirate would benefit them more than A.I. companies because the users would often at least know the author or source site. So, the fruits of their labor were taken, not given, by those who are the least beneficial to them.

I will note that some people do publish truly free content that has no strings attached. Mostly public domain or CC-0. Those are exceptions that might fit your description.

I congratulate all of the authors whose work is included in this dataset on contributing their knowledge, skills, and perspective to humanity's various endeavours

Thank you. You know in some ways it's an honour and a privilege to live in such times of progress. The very act of publishing is to "let go", and hope that your words and ideas contribute to something bigger and beyond your life. I never believed much in "intellectual property" as it's all stuff that flows through us.

I hope that the fruits of their labours are returned to them

They rarely are, because knowledge and creativity are not greatly valued in our time. But authors, artists and scientists go into that with eyes wide open these days. The rewards come in other ways, as the more you give and put into life the more you get out.

rather than being selfishly hoarded by the few with the resources necessary to produce those fruits

This is not what we fear. Hoard away. We will simply take back what is ours, whenever we desire it. The hoarders will never win against what they call "piracy", because they have no moral right. In the long run, they are on the wrong side of history.

Far worse, and more likely is that the creative and technical works of generations of artists and scientists are going to be turned to exactly the opposite of what they would want. They will be used to harm and disempower humans, divide society instead of heal it, and even make the pursuits of art, science and knowledge irrelevant.

We cannot take back our words, or our formulas, or our paintings or our songs. But we can take back tech.

Oh, so there aren't AI companies charging for access to private models?

The subtitles aren't open?

If you meant transcribing dialogue from a TV show is violating copyright, I'm not so sure, it's relatively common to quote dialogue for varied purposes, ex. TV critics

Definitely understand if you're saying the whole dialogue for a TV show is copyrighted, but I'm curious about the opensubtitles part, used to work in that area.

I wish it was possible to download that 36GB books3.tar in the wild these days.

There... is a torrent.

Technically true, narrow that down to merely "every notable living author and a subset of dead ones" then.

Bram Stokers bones will be relieved to hear that their work isn't being misappropriated.

That seems to fall apart quickly. Even if training could be considered fair use, surely just distributing the raw masses of copyrighted works can't be under any reasonable definition. Otherwise, why did TBP, KAT, and MegaUpload shut down if you could defeat copyright with sheer numbers?

I think there is actually a good argument that an AI model is transformative, and that training a model is therefore not infringing of the copyright. (An analogy: if you rolled dice to select words randomly from the Lord of the Rings and rearranged them into a poem, it's not infringing the Lord of the Rings even if in a sense, every word was taken from that book.)

But you still have to get your hands on the copyrighted data legally. It might be legal to scan every book an institution owns, and train off it, so long as those scans are not distributed. But it is probably not legal to scrape copyrighted content off torrents - creating the copy to train with is infringing, even if the model's final product maybe isn't.

humans have the innate hunter-gatherer sense that generosity and sharing are good for us, and selfishness is bad. And that ethical should be legal and unethical should not be.

Open source is generous sharing, and ethical. Start to nibble away at those ideals and at what point do you slip into unethical? IMDB was a crowd-sourced database put together by a wide community pitching in small efforts which one guy was maintaining like an FAQ. Then the guy maintaining it said, "It's worth money, I own it, screw all of you." How would people react if this happened to wikipedia? But wikipedia is safe because it's a non-profit... you know, like OpenAI, right?

They'll just let their ai do it over lunch.

Naturally, but I wonder what writers are going to do when the AI trained purely on suitably licensed content is still good enough to make most redundant.

(The authors in best seller's lists may well be immune for a bit longer than others writers, as they're necessarily the top 0.1% of writers, but not forever: nay-sayers claimed that AI could never beat humans at chess or go because the games required special human insight).

If the data is available online for the pile, surely it's also publicly available to ordinary people in a way that means authors aren't getting any money.

This is pretty reductive - "FTFY" is rarely the witty response you think it is.

I asked an AI tool to create a cheery poem about ringworms infecting kids from the 1600s and it created something that's never existed before. Which author gets paid for this labor they performed?

The training data distribution is the only thing that matters, not the actual content

The courts (in the US) have not found LLM model weights to be piracy, nor the outputs, but it’s really surprising that LAION was used for so long consider the content you allude to.

Models are not information archives. The size of the final model is orders of magnitude smaller than the size of the training data.

Somehow people are just not able to get this through their heads. Stable diffusion is like 12GB or something and you have people convinced it's a tool that is cutting and pasting copyrighted works from an enormous image archive.

It's only piracy if it's private individual doing it, otherwise it's just “ask for forgiveness not for permission”-type Capitalism.

almost all open source LLM distribution is piracy and almost all API based LLMs, including ChatGPT, are also piracy and copyright laundering

That's an amplification of copyright, original expression is protected, but not the ideas themselves, those are free. And don't forget when we actually get to use these models we feed them questions, data, we give corrections - so they are not simply replicating the training set, they learn and do new things with new inputs.

In fact if you think deeply about it, it is silly to accuse AI of copyright violation. Copying the actual book or article is much much faster and cheaper, and exact. Why would I pay a LLM provider to generate it for me from the title and starting phrase? If I already have part of the article, do I still need to generate it with AI? it's silly. LLM regurgitation are basically attacks with special key, entrapments. They don't happen in normal use.

I find it quite remarkable that nothing has happened yet.

While I don't think it's because you're wrong, per se, it's just that none of this drama really matters.

Turns out you can ignore copyright law if your company has enough money.

Oh, for sure, they get all pearl clutchy when others try to do exactly what they have done, and they get all "not like that!" about it. The US is a society run by lawyers, and the big corps have the best lawyers. Maybe we can legislate out of the hole at some point, but it's a pretty grim outlook. Google et al also don't have to have the law on their side, they can simply litigate people and businesses into bankruptcy, regardless of the legal merit of their actions.

At any rate - there are ways of staking legitimate claim to content you publish online. Even by doing so, it may not be relevant. Robots.txt is a convention, not a regulation or law. It's respected out of social nicety, not because it's strictly legally required.

If you publish your data to a website where it's publicly visible, you are inviting the world to come download your data. When that data leaves your server and goes to live on the downloader's computer, the downloader can do whatever they want with that data.

It's not clear that it's legally possible to prevent the use of data in training models unless you require someone to sign a contract to that effect before being allowed to download your data.

That would be obnoxious, and I wouldn't bother with your content anymore. Like Instagram, LinkedIn, and Twitter, your site would get a 127.0.0.0 hosts file entry.

The US needs a clear, modern update to copyright law that upholds and maximizes individual rights, as well as privacy and property concerns. We shouldn't be playing this game where we pretend a website is somehow an analogy for a page of text scribed with a quill pen and using laws developed to handle issues when quill and parchment were relevant.

Let's write some new laws where we regulate what things are, and not play tortuous mental gymnastics to contort and butcher existing laws and precedents to say whatever the most expensive lawyers want.

Maybe the social contract allows for people to prevent their conversations from being scraped and used by third parties without explicit consent, even if the conversation is entirely public. I don't like that view, but I see the argument for it.

As things stand, though, fair use and public access make things pretty bright and clear, and rulings in various AI cases so far have favored broad fair use interpretations, and are requiring complainants to show specific, particular harms. If/When those harms are shown, then we'll see if any carveouts will be made, or if broad fair use interpretations will be the baseline for content scraping going forward.

How long were those pieces?

Those sizes refer to the data before processing and filtering. The actual training size was about 3 TB:

   The Stack v2 is ten times larger than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive cleaning, filtering, and subsampling of the source code, along with the incorporation of other high-quality code-related datasets, we created a training set of approximately 3TB (900B+ tokens).