StackRating

Correlation between Reputation and “Skill”

Ever since I joined Stack Overflow I’ve been curious about whether or not there is any correlation at all between skill and reputation. For instance: Is the distribution of skill roughly the same for the top 500 users as for 500 random users of around 10k rep?

I thought long about how to estimate a users skill in a meaningful way. Accept rate and average score per answer came to mind, but neither felt very reliable. It suddenly struck me that the Elo rating system was pretty much a perfect fit. The rating system can be applied to questions ("games") whose outcome is determined by votes, and the ratings converge to values that reflect actual ability to provide good answers.

I started by analyzing the most recent data dump and proceeded to download the missing data through the StackExchange REST API. When the rating of each user was computed (refer to How the rating is computed and How to interpret the rating for details) I created a rating/reputation plot. I present to you the result of my last days of hacking and number crunching:

It’s quite easy to read and understand the plot, but I find it hand to draw any interesting conclusions. At first sight you can see that high rep users are typically also skilled users. While this is interesting in it self, one has to keep in mind that it takes a while to work up a good rating (due to the stability of the system) and to figure out Stack Overflow specific-tactics, so users with low rating can very well be just as skilled programmers.