GPT-2 and "too dangerous to release"

Read the unicorn sample if you haven’t. Someone at OpenAI gave their new language model, GPT-2, a two-sentence prompt about scientists discovering a herd of unicorns in a remote valley in the Andes, and it wrote a full news story, with a named scientist, quotes, a theory about where the unicorns came from, all of it. It’s not perfect (it calls them four-horned unicorns, and OpenAI says it took them ten tries to get that one), but it’s the first machine-generated text I’ve read that I wouldn’t immediately clock as machine-generated if it turned up in my feed.

The model has 1.5 billion parameters and was trained on about 40GB of text, around 8 million web pages that people had linked to from Reddit posts with at least 3 karma, which is a clever cheap way to get a human filter on your training data. And the headline everyone ran with last week: OpenAI isn’t releasing it. They’ve published a paper and a much smaller version of the model (117 million parameters), and they’re holding back the full one, the dataset and the training code. Their words: “Due to our concerns about malicious applications of the technology, we are not releasing the trained model.”

The ML people on my Twitter are split, and a lot of them are pretty angry. The complaints are that it’s a publicity stunt, that it goes against how the field has worked, and that it won’t actually stop anyone. I think the last one is mostly right. The paper describes the approach in enough detail, and the main ingredient is money for compute. Some well-funded lab or government could rebuild this in months. OpenAI calls it an experiment, and they do say they aren’t sure it’s the right decision, which I give them credit for. I’m just not convinced holding it back buys more than a little time.

Fake news gets the headlines, but the part that actually worries me is the signup form.

We run a SaaS product with a free tier, so we fight spam all the time, mostly fake signups that exist to post links. A surprising amount of our filtering, and I think everyone’s, quietly depends on spam being bad writing. The same template with the nouns swapped out, the grammar a bit off. We have a rule that flags near-duplicate text across new accounts and it catches a ridiculous share of what gets through. That rule is worth nothing against a model that can write a different, fluent, on-topic paragraph every time. The same goes for fake product reviews, and for the phishing email that used to give itself away in the second sentence.

None of that needs the full 1.5B model, honestly. The small one they did release is probably already a lot better than what spammers use today. So the defenses have to move off “does this text look human” and onto things that are harder to generate: account age, behavior, payment, network reputation, rate limits. Most of that is unglamorous plumbing we should have anyway.*

I’ll be downloading the small model this weekend to see what it does with our own help-center articles as prompts. If it’s good at writing plausible support requests, I’d like to know that before someone else figures it out.

* Also, if you’ve been meaning to put a real CAPTCHA on a form and have been putting it off because the spam was manageable, the math on that may be about to change.