What Jaap Stam can tell us about education data.
Yes I know, start with the mass market stuff to grab the readers then, once they are hooked, start on the meaty topics.
In the many hours I waste on the internet when I should be doing whatever it is that people who don't spend hours on the internet do (work is something I've heard mentioned) I like to imagine how I'd be much better at other people's jobs. I then supply them with handy tips about how they might get better at what they do, safe in the knowledge that I'll never have to actually do the work, and that anything I recommend, that isn't done, would have worked perfectly.
With this in mind I want to write some quick thoughts down on two areas that interest me; football and education.
Both areas seem to be in the middle of an analytics revolution. Or as Mark Lawrensen would no doubt describe it "attack of the nerds". Nerd being, in football circles, someone who thinks there is any value at all in looking at how things could be improved who has never been paid to kick a ball.
The early days of football analytics are filled with, perhaps exaggerated, stories of the success or, usually failure, of "the men with laptops". Perhaps the most famous story of all is Sir Alex Ferguson being persuaded, against the advice of his own eyes, to sell his main defender Jaap Stam as his tackling stats had fallen away dramatically. The "nerds" convinced him that he was in a sudden decline and that the club should accept a high offer from the Italian club Lazio. So they accepted the bid and Stam went on to continue his excellent level of performance for years to come. The data they had measured a decline in tackles but there was no measurement at the time for interceptions. Rather than waiting for an opponent to get the ball and tackle him Stam was stepping forward and intercepting the pass. Eyes 1, Nerds 0.
It was many years before sufficient data was captured for there to be anything more than superficial analysis done of football performance. And even then the caveats that almost every statistic presented have to come with have made single figure metrics in football difficult to take seriously. Yes I can see that this player makes more passes per game than this player. But are the passes better passes? Do they create goalscoring opportunities? Or are they simply playing lots of backward and sideways passes that are safe but ultimately pointless? Who are they playing alongside? A team of superstars will make an average player look a lot better thanks to their superior movement. Put me alongside 9 Lionel Messi's and a goalkeeper and we'd win the league.
One of the most interesting recent findings was that, essentially, if a basic level of competence is reached, then it doesn't really matter who the manager of a team is. Like in everything there are outliers in both directions, and in some cases a manager regarded as outperforming the masses (Moyes at Everton) moves clubs and it dragged back into the mix (Moyes's subsequent career)
Almost all of the difference in the level of performance of a football team is down to their payroll. Paying me £10m a year won't make me a decent footballer, but in the free market that is Premier League, football players generally find their level. Overtime the best paying clubs attract the best talent and that talent wins them trophies.
Which moves me on to schools. We are often treated to stories in the press about how a superhead has turned around a school and how this school with it's insistence on kids wearing blazers and ties in 40c heat is justified by the amazing exam results.
Yet time and again we find the schools with the most success have some external factor that needs to be considered. It may be they have a high achieving demographic of pupils (Chinese female pupils joining with English as a second language seem to be the dream value added candidates), a much larger than average amount of money per pupil to offer additional support, an extremely harsh, high achieving reputation which effectively selects out only the brightest and hardest working children with supportive families. Schools which achieve excellent results, with happy staff and well rounded pupils, without these factors are the true outliers and should be applauded. But there aren't many, and that is OK, that is the definition of being an outlier.
We then have the use, and misuse of research in both sectors. Every passing fad from brain gyms, to thinking hats and growth mindset (cognitive load coming soon!) is evidence based to start with. A small scale study on a population subset, lead by enthusiastic experts shows a better than average outcome for those following the plan. It would be a crime against children not to rush out these schemes that are scientifically proven to work. And sell Jaap Stam to Lazio whilst you are at it.
You see the problem is always small sample sizes and single points of data. Stam showed a decline in one metric, but another unmeasured metric was rising unseen at the same time. Human beings are complicated things. Sometimes you need the "eye" test.
Getting the correct data to make meaningful judgments took years in football, an industry flooded with cash with billions at stake. Analysts are now part of the staff at most top clubs (though distrusted by "proper football men") Too often we see schools judged on small samples and a narrow range of measurements. We need to stop and step back before we use data to condemn good people doing the right thing and adding value in ways we can't yet measure.
In the many hours I waste on the internet when I should be doing whatever it is that people who don't spend hours on the internet do (work is something I've heard mentioned) I like to imagine how I'd be much better at other people's jobs. I then supply them with handy tips about how they might get better at what they do, safe in the knowledge that I'll never have to actually do the work, and that anything I recommend, that isn't done, would have worked perfectly.
With this in mind I want to write some quick thoughts down on two areas that interest me; football and education.
Both areas seem to be in the middle of an analytics revolution. Or as Mark Lawrensen would no doubt describe it "attack of the nerds". Nerd being, in football circles, someone who thinks there is any value at all in looking at how things could be improved who has never been paid to kick a ball.
The early days of football analytics are filled with, perhaps exaggerated, stories of the success or, usually failure, of "the men with laptops". Perhaps the most famous story of all is Sir Alex Ferguson being persuaded, against the advice of his own eyes, to sell his main defender Jaap Stam as his tackling stats had fallen away dramatically. The "nerds" convinced him that he was in a sudden decline and that the club should accept a high offer from the Italian club Lazio. So they accepted the bid and Stam went on to continue his excellent level of performance for years to come. The data they had measured a decline in tackles but there was no measurement at the time for interceptions. Rather than waiting for an opponent to get the ball and tackle him Stam was stepping forward and intercepting the pass. Eyes 1, Nerds 0.
It was many years before sufficient data was captured for there to be anything more than superficial analysis done of football performance. And even then the caveats that almost every statistic presented have to come with have made single figure metrics in football difficult to take seriously. Yes I can see that this player makes more passes per game than this player. But are the passes better passes? Do they create goalscoring opportunities? Or are they simply playing lots of backward and sideways passes that are safe but ultimately pointless? Who are they playing alongside? A team of superstars will make an average player look a lot better thanks to their superior movement. Put me alongside 9 Lionel Messi's and a goalkeeper and we'd win the league.
One of the most interesting recent findings was that, essentially, if a basic level of competence is reached, then it doesn't really matter who the manager of a team is. Like in everything there are outliers in both directions, and in some cases a manager regarded as outperforming the masses (Moyes at Everton) moves clubs and it dragged back into the mix (Moyes's subsequent career)
Almost all of the difference in the level of performance of a football team is down to their payroll. Paying me £10m a year won't make me a decent footballer, but in the free market that is Premier League, football players generally find their level. Overtime the best paying clubs attract the best talent and that talent wins them trophies.
Which moves me on to schools. We are often treated to stories in the press about how a superhead has turned around a school and how this school with it's insistence on kids wearing blazers and ties in 40c heat is justified by the amazing exam results.
Yet time and again we find the schools with the most success have some external factor that needs to be considered. It may be they have a high achieving demographic of pupils (Chinese female pupils joining with English as a second language seem to be the dream value added candidates), a much larger than average amount of money per pupil to offer additional support, an extremely harsh, high achieving reputation which effectively selects out only the brightest and hardest working children with supportive families. Schools which achieve excellent results, with happy staff and well rounded pupils, without these factors are the true outliers and should be applauded. But there aren't many, and that is OK, that is the definition of being an outlier.
We then have the use, and misuse of research in both sectors. Every passing fad from brain gyms, to thinking hats and growth mindset (cognitive load coming soon!) is evidence based to start with. A small scale study on a population subset, lead by enthusiastic experts shows a better than average outcome for those following the plan. It would be a crime against children not to rush out these schemes that are scientifically proven to work. And sell Jaap Stam to Lazio whilst you are at it.
You see the problem is always small sample sizes and single points of data. Stam showed a decline in one metric, but another unmeasured metric was rising unseen at the same time. Human beings are complicated things. Sometimes you need the "eye" test.
Getting the correct data to make meaningful judgments took years in football, an industry flooded with cash with billions at stake. Analysts are now part of the staff at most top clubs (though distrusted by "proper football men") Too often we see schools judged on small samples and a narrow range of measurements. We need to stop and step back before we use data to condemn good people doing the right thing and adding value in ways we can't yet measure.
Comments
Post a Comment