A couple of days ago, I came across this fine paper by Keir Mierle et al. from U of Toronto:
Mining Student CVS Repositories for Performance Indicators
They examined the CVS repositories that their students used for assignments, extracted all sorts of numbers by analysing access patterns and code metrics, and correlated the numbers with the grades the students got on the assignments.
Basically, they didn’t find any good indicators. The best indicator they found is raw lines of code. Unsurprisingly, hard-working students got better grades. The correlation isn’t very strong though.
The second-best indicator: Did the students put spaces after commas in their sourcecode? Students who write foo(a, b, c) tend to get better grades than students who write foo(a,b,c).
The authors conclude:
Although version control repositories contain a wealth of detailed information both in the transaction histories and in the actual files modified by the users, we were unable to find any measurements in the hundreds we examined which accurately predicted student performance as measured by final course grades; certainly
no predictor stronger than simple lines-of-code-written was found.
>
These results directly challenge the conventional wisdom that a repository contains easily extractable predictive information about external performance measures. In fact, our results suggest that aspects such as student work habits, and even code quality, have little bearing on the student’s performance.
This doesn’t bode well for tools like StatCVS. We’re certainly generating interesting information, but this paper provides some evidence that we can’t measure performance.