Writing our AI usage policy

We’ve had a written policy for Copilot since last July, a few lines long. The first draft of our policy for everything else was one sentence: “Don’t paste anything into ChatGPT that you wouldn’t paste into a public GitHub issue.” I wrote it in about thirty seconds, in March, after a week of GPT-4 and three people asking me whether they were allowed to use it at work. It’s still the best sentence in the document, and the document is now a page and a half, because it turned out the one sentence answered maybe a third of the questions people actually had.

I’m writing up what we ended up with because two other people running engineering teams have asked me for it in the last fortnight, and because I suspect most small companies are in the same spot: everyone’s using these tools already, nobody’s written anything down, and the only real policies out there are bans from big companies, which aren’t much use to a company our size.

Why not ban it

We talked about it, briefly. The argument for a ban was mostly the news. Samsung’s engineers were reported earlier this month to have pasted source code and meeting notes into ChatGPT, and Samsung has reportedly put limits in place since. Italy’s data protection authority ordered OpenAI to stop processing Italians’ data at the end of March. And on March 20th ChatGPT itself had a bug that showed some users the titles of other people’s conversations, and for a small number of subscribers, some of their payment details, which OpenAI wrote up a few days later and traced to a bug in an open-source library.

None of that convinced me a ban would work. I know from my own text file that I’m getting real time back from these tools, and I know at least half the team has been using ChatGPT since December whether or not anyone told them they could. A ban would mean that keeps happening, just without anyone telling me.

What’s allowed, with what

The policy is organized by tool, because that’s how people think about it. “Can I use X for Y” is the question, never “what is our data classification for prompts”.

GitHub Copilot is allowed for all engineers, on the company’s Copilot for Business account. Since last July people have used their own individual subscriptions, reimbursed by the company, with the public-code filter on and not in two repos, the billing service and the one with our authentication code. I was comfortable enough with that, but it was our code going out under each person’s personal terms. The business plan launched in February, says it doesn’t keep the code snippets it sees, and lets us turn on the filter that blocks suggestions matching public code, which we’ve done. It’s $19 a person a month, and nobody argued about that. The two excluded repos stay excluded. The business terms are better, but I still don’t want to have the auditor conversation about them yet.

The OpenAI API is allowed, through the company account, for internal tools and experiments. Since the start of March, OpenAI says it doesn’t train on data sent through the API unless you opt in, and keeps it for 30 days for abuse monitoring. That’s a contract I can read and point at, which is the difference between it and the consumer product.

ChatGPT, the website, is allowed for anything you’d be fine seeing in public. General programming questions, explaining an error message, drafting a blog post, rewriting your own badly worded email. Conversations in the consumer product can be used to train their models, and that’s the one place the one-sentence rule still does all the work.

Everything else (the other chatbots, browser extensions, the AI features every SaaS vendor we use has bolted on this quarter) needs to be asked about first, in a channel, so that the answer is public and the next person can find it.

What’s never allowed, anywhere

This is the short list, and it’s the part I care about most:

• customer data of any kind, including “anonymized” exports, which in my experience are never as anonymous as the person exporting them thinks
• credentials, keys, tokens and connection strings, even expired ones
• anything covered by a customer contract that says it stays with us, which is more of our contracts than people realize
• other people’s personal information, which includes candidate resumes and interview notes

We had an argument about source code. One position was that our code is a trade secret and shouldn’t go to any third party, Copilot included. The other was that it already goes to GitHub, our CI provider and a dozen other vendors, and Copilot for Business is the same kind of contract. I sided with the second, with the exception that code goes only to the tools on the allowed list and never to the consumer ChatGPT, and I’m not completely sure I’m right.

What we left out on purpose

We don’t require anyone to label AI-generated code in pull requests. I went back and forth on this. It would be nice to know. But it’s unenforceable, it’d be wrong half the time (is a Copilot completion I rewrote “AI-generated”?), and the moment you require it, people stop mentioning it voluntarily, which they currently do. What the policy says instead is that you own every line you submit, however you got it, and review treats it the same. That was already true. It just hadn’t been written down.

We don’t have a rule about customer-facing features built on these models yet, because we haven’t built one. What the policy says is that nothing goes to a customer that a model wrote and a person didn’t read, and that before any feature sends customer data to a model provider, that provider goes through the same security review as any other vendor, and onto the subprocessor list in our customer agreements, which means telling customers first. That second part is going to slow us down when we do get to it, and it should.

And we allow candidates to use these tools in our work sample, the one-hour exercise we run over video. That one surprised some people. My reasoning is that the work sample is supposed to look like the job, and the job now includes them. We ask people to tell us what they used and how, and the follow-up conversation about the code is where we find out whether they understand it. Two candidates so far have told us. Both said it helped with the boilerplate and got the tricky part wrong, which is about what I’d have predicted.

The section of the policy I’m least happy with is “the AI features every SaaS vendor we use has bolted on”. Our wiki has an AI assistant now. So does our meeting tool, which offered last week to summarize a call for me that included a candidate’s salary expectations. Our support desk has one. Each of those sends our data to a model provider under that vendor’s contract, not ours, and in most cases I haven’t been able to find out which provider or under what terms, and turning them off is a different setting in each product, sometimes per user. I’ve asked our security lead to go through the vendor list and check every one, and the column he added to the spreadsheet is called “AI?”, and so far it has more question marks in it than answers.