Peter Cowley’s Flawed Logic
From the media (link), we learned, Peter Cowley, one of the authors of an annual report from the Fraser Institute, hinted that he observed seemingly consistent gaps between school grades and diploma grades and concluded, in part, that students are being cheated by teachers or the system.
“In essence, students are being cheated by allowing them to think that they understand the material better than they actually do.”
– Peter Cowley
He singles out teacher’s or the education system’s input as the sole culprit. We spent an hour reading thru his report carefully, picking out important information and tried to find proof such correlation. We could not. Not only did the report indicate an absence of evidence (i.e. no evidence to support his claim), the data, at least in part, suggests strong evidence of absence (i.e. enough evidence to prove Mr. Cowley’s claim is false) The report failed to present logical arguments to support his claim. The existence of a gap is not a proof itself for anything but a theory – a gap is observed, and we look for plausible explanations. We hypothesize a theory to explain the observable gap, not using the gap to explain the hypothesis. To practice science, one must allow a mechanism for falsification. It’s our attempt to falsify Mr. Cowley’s claim.
School Grades and Exam Grades are Inherently Different
After examining the data and use simple logic, we reject Mr. Cowley’s theory that students are being cheated. The gap itself can be a product of a number of reasons. One of which is the nature of the two systems. The School grade is comparable to an Income Statement of a firm. It shows the infusion of knowledge (teacher driven) and effort (student driven) between two periods, the beginning and the end of a course. Reported school mark is a weighted average of all the course components combined, from the beginning to the end. The Diploma exam mark, however, can be compared with a Statement of Financial Position (aka balance sheet); it gives a reading of accumulated knowledge or wealth of knowledge at a given point in time. It’s like a snapshot or an X-ray. The two measures are meant to measure different outcomes. Why would we assume they are equal as a reference point? School mark represents the journey from X to Y, while exam mark reflects accumulated achievement at the time of test writing. We argue they are inherently unequal, hence the reference should be that they are unequal. On this note, we think Mr. Cowley is wrong. Provided the exams are fairly consistent in terms of difficulties, a growing trend is more worrisome than a consistent difference. A consistent gap is, in our point of view, normal, likely the result of a stochastic process or randomness (lack of information or knowledge). Randomness, by definition, does not lend itself to any explanation. A collection of random events behave in a highly regular fashion. It’s the nature of statistics.
School and Diploma Gap
In Alberta, two separate components, School and Exam grades, help determine what’s considered to be the final mark. Note that the weighing of the exam changed as of September 2015, from 50/50 to 70/30. The former suggests equal weighting between School Grades and Diploma Exam Grades, the latter signals a 70% reliance on School grades.
Keep in mind diploma exam result is a proxy of student ability at the time of writing (x-ray) which is influenced by the following factors (not in any particular order):
- Teacher’s input
- School’s input
- School’s policy
- Exam consistency across years
- Student’s effort
- Format difference
- Environmental factors (catastrophic events)
- Unknowns (unobserved elements, such as illness, measurement or grading errors)
The following graphs and figures are calculated and constructed using data downloaded directly from Alberta Education. We focus mainly on Average School Mark (SchAvg) and Diploma Exam Mark (ExamAvg). There are a number of statistical tests we can conduct at a later time, parametric or non-parametric alike of which we chose to exclude. As mentioned before, the gap itself is NOT a proof teachers are cheating students. By symmetry, the absence of gap does not prove teachers are NOT cheating students. We just have no proof cheating exists empirically, which is often referred to as absence of evidence.
In conclusion, we are unable to find any proof that students being cheated by educators or the system in the dataset. In addition, Mr. Cowley’s logic is seriously flawed. We observed growing gaps in Math 30-1 and Math 30-2. Here, we present a logical and plausible alternative explanation.
*2012 data Not Available from Alberta Education
*2012 data Not Available from Alberta Education
|Social Studies 30-1||2012||2013||2014||2015||2016||2017|
|Social Studies 30-2||2012||2013||2014||2015||2016||2017|