R for a C# programmer

var totals = orders
    .Where(o => o.Year == 2018)
    .GroupBy(o => o.Region)
    .Select(g => new { Region = g.Key, Total = g.Sum(o => o.Amount) })
    .OrderByDescending(r => r.Total);

totals <- orders %>%
  filter(year == 2018) %>%
  group_by(region) %>%
  summarise(total = sum(amount)) %>%
  arrange(desc(total))

The first one is what I’d have written for the last ten years. The second is what I write now, most weeks, because I’ve spent a good part of the past year doing a data science certificate in R. The day I saw dplyr written like that was the day R stopped feeling like a hostile foreign country, because that’s LINQ. It’s the same idea, a pipeline of small verbs over a sequence of records, and even the verbs line up: filter is Where, group_by is GroupBy, summarise is the Select over groups, arrange is OrderBy. If you’re a C# programmer being made to learn R, start there.

What the tidyverse got right

The tidyverse is Hadley Wickham’s (and a lot of other people’s) set of packages, dplyr, tidyr, readr, ggplot2 and friends, that share a design. Before I found it, R was base R to me: square brackets, apply functions and $ signs, and a lot of Stack Overflow.

The thing it gets most right is that it’s built around one data shape. “Tidy data” means each variable is a column, each observation is a row, and every function takes a data frame as its first argument and returns one. That’s what makes the %>% pipe work, since the output of each step is always a valid input to the next. In C# terms it’s as if every LINQ operator took and returned the same type, and the type happened to be a table. I spent years in .NET getting data into the right shape for whatever I wanted to do next. In the tidyverse the shape is always the same, and tidyr::gather() and spread() exist for the times it isn’t.

ggplot2 is the other one. It’s based on Leland Wilkinson’s grammar of graphics, and instead of picking a chart type from a menu (“bar chart”, “line chart”), you describe a mapping from columns to visual properties, then add layers:

ggplot(monthly, aes(x = month, y = total, color = region)) +
  geom_line() +
  geom_point() +
  facet_wrap(~ year)

Map month to x, total to y, region to color, draw lines and points, and give me one small chart per year. Every time I’ve wanted to change something, it’s been one more + and not a different API. I’ve used a lot of charting libraries in C# and JavaScript over the years and none of them came close. I now prototype charts for work in ggplot2 before building them anywhere else, which I would not have predicted a year ago.

RStudio deserves a mention too. It’s the closest thing R has to Visual Studio, and R Markdown, where the code, its output and your prose live in one document, is how every one of my assignments gets written now.

What still drives me mad

Indexing starts at 1. I know, I know. I’ve had a year. I still get it wrong about once a week, and the fun part is that x[0] quietly returns an empty vector instead of throwing, so the bug shows up three steps later.

Vectors recycle, silently:

> c(1, 2, 3, 4) + c(10, 20)
[1] 11 22 13 24

The shorter one just gets repeated to match the length of the longer one. That’s deliberate and occasionally useful (adding a scalar to a vector is the same rule), and you only get a warning if the lengths don’t divide evenly. Coming from a language where a length mismatch is an exception, this is horrifying. I lost an evening to it in the spring: I was comparing 12 monthly targets against what I thought was 12 months of actuals for one region, except a filter I’d forgotten about meant it was 24 rows for two regions. No warning, because 24 divides by 12, just a set of numbers that looked plausible and were wrong.

Then factors. Until you tell it not to, read.csv() turns any column of strings into a factor, R’s categorical type, which is stored as integer codes with a table of labels. Now suppose a numeric column has one stray "n/a" in it. The whole column becomes a factor, and as.numeric() on it gives you the codes, not the numbers. No error, just wrong numbers. The fix is stringsAsFactors = FALSE everywhere, or readr’s read_csv(), which doesn’t do this, and which I now use exclusively.

There’s no compiler to save you. T and F mean TRUE and FALSE until someone writes T <- 0. sapply returns a vector, or a matrix, or a list, depending on what your function happened to return. Dots in names (is.na, data.frame, t.test) look like method calls and mostly aren’t, except when they are, because S3 dispatch uses the dot (print.lm is print for lm objects). And there are at least three object systems, S3, S4 and Reference Classes, plus the popular R6 package. As a C# person I expected types to tell me what I was holding. In R I end up calling str() on everything.

And finally the thing that makes the tidyverse so pleasant to use is what makes it unpleasant to write functions around. In filter(year == 2018), year isn’t a variable in scope, it’s a column name that dplyr captures unevaluated and looks up in the data frame. That’s non-standard evaluation, and it’s great right up until you want to pass the column name in as a parameter:

total_by <- function(df, col) {
  col <- enquo(col)
  df %>%
    group_by(!!col) %>%
    summarise(total = sum(amount))
}

enquo captures the argument as a quosure and !! unquotes it back in. It works, and I understand it now, roughly, but it took me a weekend and I’d struggle to explain it to someone on my team without a whiteboard. It’s the most C#-hostile idea in the language, because in C# an expression is either code or an Expression<T> tree and the compiler tells you which. Here it’s whatever the function you’re calling decides.

Would I use it at work?

For analysis, yes, and I already do. When someone asks a question about our data, an R Markdown file with a few dplyr pipelines and a ggplot is faster than anything else I’ve got, and I can send the knitted HTML to people who’ve never heard of R. For anything that has to run every day in production, no. That stays in a language with a compiler, and I’ll keep the R notebook next to it as the spec.