Things I've learned in nearly 30 years of programming

The first program I wrote that someone else used was a Clipper app on DOS, around 1992, when I was 13. It kept a list of customers and printed invoices on a dot-matrix printer that you could hear from the next room. I wrote it in a weekend, and I spent the next three months fixing it, because I’d assumed every customer name would fit in 20 characters. That’s about the most useful single lesson I’ve ever learned, and it’s taken me most of 30 years to fully absorb.

Justin Etheredge’s 20 things I’ve learned in my 20 years as a software engineer went around Hacker News last month, and I agreed with most of it, especially “I still don’t know very much” and the line about avoiding 0.1x programmers rather than hunting 10x ones. I’ve been meaning to write my own version for a while. It’s not 20 items, and some of them contradict things I believed loudly on this blog years ago, which seems like the right way to do it.

Your assumptions about data are wrong, and the data will tell you eventually. The 20-character name field. Dates in another time zone. A “unique” email that two customers share. Almost every production bug I’ve caused came down to something I believed about the input that the input didn’t agree with. I now ask “what’s the biggest one of these we’ve ever seen?” before I write any code that handles a list, a string or a file, because the answer is always bigger than I’d have guessed.
Nearly every performance problem I’ve fixed was one loop. Usually a loop inside a loop, hidden in a library call. I wrote about the GTA Online loader in March, and it’s the same story every time: someone measures, and it turns out not to be the database, the language or the framework, it’s a linear search somebody wrote years ago with ten rows of test data. Measure before you guess. I still guess first and I’m still usually wrong.
Tools matter less than I thought, and then more. In 2009 I wrote a post saying tools don’t matter, only the mind using them. I half agree with that guy. Your editor really doesn’t matter. Your debugger, your profiler and your ability to reproduce production locally matter enormously, and I’ve watched a team double its speed by getting one of those right. The difference is whether the tool changes what you can see.
You can’t secure, patch or migrate what you don’t know you have. I’ve written versions of this after Shellshock, after leaving the colo, after a Struts jar nobody remembered. The inventory is always the project. The policy is easy to write and only covers the things on the list.
Write it down, and then write down why. Design docs, postmortems, the comment above the weird line. The why is the part that’s gone in two years. I can read code to find out what it does, but nothing in the code tells me that the strange retry loop is there because a payment provider used to return 200 with an error in the body. Half the bad decisions I’ve seen were someone removing something whose reason had been forgotten.
Being wrong in public is cheap, and being wrong in private is expensive. In college I told an online forum that the professor teaching us Python was an idiot pushing a “typeless” toy language. That one’s on this blog too. It cost me nothing except some embarrassment later. The expensive mistakes have all been the ones I kept to myself, a design I had doubts about and didn’t raise, a hire I was unsure of and didn’t push back on, because saying so felt awkward.
The hard part is almost always the seams. Every big project I’ve seen go wrong had components that worked on their own and broke when you put them together, which was the Healthcare.gov lesson and is also the lesson of every microservice migration I’ve been near. Integrate early, deploy the skeleton on day one, and be suspicious of anyone reporting that their piece is 90% done.
The best interview I can run looks like the job. I gave up puzzles years ago. Real code to read, a real bug, a small thing to build. I’m still not great at predicting who’ll be good, and I don’t think anyone is, but at least now I’m wrong about things that matter for the work.
Management is a different job. I moved into it about fifteen years ago and I’d make the same call again, but I was bad at it for the first two years and nobody told me, because I was their manager. If you’re going to do it, find someone outside your reporting line who’ll tell you the truth.
Choose technology your team can debug at 3am. Not the one that’s best on paper. The one where, when it breaks at 3am, the person on call has seen it break before and knows where the logs are. That’s usually the older, less exciting option, and I’ve had to relearn this one more than once.
Assume every secret leaks eventually. Passwords, API keys, the vault itself. Design so that a leaked credential is a bad day rather than a catastrophe, which mostly means 2FA, short-lived credentials and roles that can do one thing. Every incident I’ve written about here got smaller wherever someone had already assumed that.
You don’t stop needing to learn things, and it doesn’t get easier. I did a data science certificate a few years ago, mostly in R, and it was harder than anything I’d studied since my math degree, partly because I had a job and partly because I’d gotten used to being the person who already knew. Being a beginner again was good for me. It also made me a lot more patient with the new engineers on my team.
The work is mostly people. I didn’t believe this at 25. I’d have told you the work was code and the people were a distraction. Thirty years in, the projects that succeeded were the ones where people trusted each other enough to say “I’m stuck” early, and the ones that failed were mostly the ones where they didn’t. The technology was rarely the reason either way.
I still don’t understand networking. I’ve been doing this nearly 30 years, I was first made a Microsoft MVP back in 2005, and last month I wrote a whole post about BGP that I had to check with two people who actually understand it. There’s always a layer underneath the one you know. For me that’s been networking, and for the last few years it’s also been statistics, and I suspect in a few years it’ll be something I haven’t heard of yet.
It’s still fun. The weekend with the invoice printer was fun, and so was typing in all of Crafting Interpreters this summer. I’ve had bad years in this job, a couple of them, but I’ve never had a stretch where I didn’t want to build something.