Category Archives: Education Trainwrecks

Information Quality problems impact reporting in US Education

Leave a reply

Via The Miami Herald comes a story that highlights a number of impacts of poor quality information in key processes.

In Oklahoma, schools are placed on an “improvement list” if they fail to meet standards for two consecutive years. Once on the list the school show progress in improving standards for two years before they can be taken off the list. This can have implications for funding and access to resources as well. Some Oklahoma School districts are, it is reported, concerned that they don’t make the grade against Federal requirements.

Problems with the quality of demographic data in electronic testing performed by Pearson has affected the publication of the reports against which schools are graded.  These will now be available a full month late, being released in September and not August as expected. This will affect the ability of School Boards to effectively respond to their report card.

Other problems reported on top of missed deadlines include errors in printing report cards to be sent to parents

Oklahoma’s Superintendent of Schools Janet Barresi has described the impacts of poor quality data in this process as a “ripple effect”  that is “imposing an unacceptable burden on school districts” and has called for Pearson’s contract to be reviewed. Pearson are engaging an independent 3rd party to help verify the accuracy and validity of the scoring data (which they are confident in).

Oklahoma is not the first State where data issues have been a problem.

  • In 2010 in Florida Pearson was penalised $14.7 million, and had to ramp up staffing levels and make changes to systems as a result of problems with information quality leading to delays. The problems here related to matching of student records.
  • In 2010 in Wyoming, Pearson also had to pay penalties arising from problems with the testing, ranging from data going missing to other administrative problems such as improperly calibrated protractors.

This video from the Data Quality Campaign, a US Non-Profit working to improve standards of data quality in the US Education system, highlights the value of good quality and timely information in this important sector:

No child left behind (except for those that are)

Leave a reply

Steve Sarsfield shares with us this classic tale of IQ Trainwreck-ry from Atlanta Georgia.

An analysis of student enrollment and transfer data carried out by the Atlanta Journal-Constitution reveals a shocking number of students who appear to be dropping out of school and off the radar in Georgia.  This suggests that the dropout rate may be higher and the graduation rate lower than previously reported.

Last year, school staff marked more than 25,000 students as transferring to other Georgia public schools, but no school reported them as transferring in, the AJC’s analysis of enrollment data shows.

Analysis carried out by the State agency responsible was able to track down some of the missing students. But poor quality information makes any further tracking problematic if not impossible.

That search located 7,100 of the missing transfers in Georgia schools, state education spokesman Dana Tofig wrote in an e-mailed statement. The state does not know where an additional 19,500 went, but believes other coding errors occurred, he wrote. Some are dropouts but others are not, he said.

In a comment which should warm the hearts of Information Quality professionals everywhere, Cathy Henson, a Georgia State education law professor and former state board of education chairwoman says:

“Garbage in, garbage out.  We’re never going to solve our problems unless we have good data to drive our decisions.”

She might be interested in reading more on just that topic in Tom Redman’s book “Data Driven”.

Drop out rates consitute a significant IQ Trainwreck because:

  • Children who should be helped to better education aren’t. (They get left behind)
  • Schools are measured against Federal Standards, including drop out rates, which can affect funding
  • Political and business leaders often rely on these statistics for decision making, publicity,  and campaigning.
  • Companies consider the drop out rate when planning to locate in Georgia or elsewhere as it is an indicator of future skills pools in the area.

The article quotes Bob Wise on the implications of trying to fudge the data that sums up the impact of masking drop outs by miscoding (by accident or design):

“Entering rosy data won’t get you a bed of roses,” Wise said. “In a state like Georgia that is increasingly technologically oriented, it will get you a group of people that won’t be able to function meaningfully in the workforce.”

The article goes on to highlight yet more knockon impacts from the crummy data and poor quality information that the study showed:

  • Federal standard formulae for calculation of dropouts won’t give an accurate figure if there is mis-coding of students as “transfers” from one school to another.
  • A much touted unique student identifier has been found to be less than unique, with students often being given a new identifier in their new school
  • Inconsistencies exist in other data, for example students who were reported “removed for non-attendance” but had  zero absent days recorded against them.

Given the impact on students, the implications for school rankings and funding, the costs of correcting errors, and the scale and extent of problems uncovered, this counts as a classic IQTrainwreck.

Irish State Exam leak being studied.

Leave a reply

A serious complication has emerged in Leaving Certificate exams run by the Irish State each year.. An exam Superindtendent accidentally distributed the wrong paper in one exam centre earlier this week. He put out the exam questions for Paper 2 of the English examination, which wasn’t the subject being examined. The paper was, it seems, only on students’ desks for a few minutes before the error was noticed. However, in this age of twitter, bebo, myspace, facebook and such things, details of the exam questions were soon being discussed in school yards the length and breadth of the country. 

To make matters worse, the Superintendent in question failed to notify the Department of Education until more than 6 hours after the paper was leaked.

As with all things governmental, an investigation is underway. Denials of responsibility have issued from various entities involved.  The Superintendent in question has been dismissed. The exam is being rescheduled, causing disruption to study timetables across the land.

But an examination (no pun intended) of the facts reveals a telling IQ Trainwreck.

One of the factors that determines the quality of information is the quality of information presentation. Indtroducing ambiguity into visual information invites error. Tom Redman, in his book Data Driven, describes the presentation of information as a key step in how information is used and a key part of its complexity. Redman tells us that a number of disciplines need to come together to make even the simplest information and data useful, including:

Presenting data in ways that make it easy for customers to understand and use them. Only in this last step do data and information contribute to internal operations and decisions…

Packaging two sets of highly sensitive information in highly similar packaging which is similar enough that a warning is required makes it hard for customers (Redman uses “customers” to mean the actual consumers of the information – in this case the Superintendent) invites misunderstanding and error.

Yes, the Superintendent could have and should have double checked the paper was the right paper before handing it out, but a key contributing cause was the use of overly similar packaging for both exams.

  • The Superintendent didn’t report a leak of sensitive information in a timely manner

All too often this happens in business. A laptop gets stolen, a memory stick gets mislaid, sensitive information gets left on a train. A key element of the response to this kind of problem is knowing that there is actually a problem, so early reporting to authorities of the leak is imperative. Had the State Examinations Commission had the information in a timely manner perhaps the cost of fixing the gaffe would be less.

  • The cost of remedying the issue is now put at approximately EUR 1 Million

The solution that the Department of Education and State Examinations Commission has come up with is to run a totally new exam paper on Saturday. That means:

  • Extra costs for transport for students to the exams (where State-funded school transport is used)
  • Extra salary costs for Superintendents and their assistants
  • Extra salary costs for school staff who are required to be on-site during exams.
  • The costs of printing a whole new batch of exam papers.

And of course, it being a Saturday:

The SEC is finalising arrangements for a deferred sitting of English papers for a small number of students from the Jewish community after getting legal advice that asking them to sit an exam on their Sabbath, when their religion prohibits it and it is against their conscience, could have been unconstitutional. All other students will be expected to attend, in line with other timetabled exams.

For more on that particular complication, see the Irish Times’ detailed story.

So, why is this an IQTrainwreck?

  1. The similarity in packaging on the exam papers was a key root cause. This is (or should be) a straightforward process of ensuring that all exam subjects and levels are distinctly colour coded and ensuring that packaging is not similar. Issuing a reminder is simply trying to inspect a defect out of the process. Yes, the Superintendent has to carry responsibility as well for not double checking but avoidable similarity should have been avoided (ergo preventing the confusion)
  2. The lack of rigour regarding the reporting of the accidental distribution of the wrong paper is inexcusable. 
  3. The cost impact of the error is extremely significant, particularly given the current state of Irish Government finances. EUR1 Million is a challenging amount to find in your budget at short notice.
  4. The disruptive impact on students during a stressful time can’t be underestimated. 
  5. The further complication presented by Ireland’s multi-culturalism adds further challenges (and potentially costs) for the SEC, the Department, and the students.

(On that note of multiculturalism, one is left wondering if the ISM school in Tripoli, Libya that offers the Irish Leaving Certificate to its students will have received their replacement exam papers yet of if they are even aware of the issue.)

Dead girl given truancy warning

Leave a reply

Courtesy of #dataquality twitterers Steve Tuck and Stephen Bonner comes this story from the BBC about a school in Cheshsire whose parents received a truancy notice about their daughter which threatened to ban her from her end of year prom for being over 30% below the target attendance rate for students.

The young girl, Megan, had possibly the best excuse ever for playing hookey from school however. According to her mother:

“Megan doesn’t go to that school any more. She’s been dead for two months now so it’s not surprising her attendance is low.”

It appears that inconsistencies between two computer systems in the school resulted in the school’s left hand not knowing what the right hand was doing with regard to student information.

Megan’s name had been taken off the school roll when she died, and removed from the main school database,” the spokeswoman said.

“However, unknown to the school, her details had remained in a different part of the computer system and were called up when the school did a mail merge letter to the parents of all Year 11 students about their prom”.

Reading the comments from the software providers in the BBC story, it would also appear that the software lacks a “dead student” flag to enable them to exclude deceased students from administrative mailings.

This is a classic IQTrainwreck because it resulted in distress and upset to Megan’s parents, landed on the BBC News website (with video no less) , has been flashed across Twitter, and has now wound up here.

Also, this failure of the computer systems to allow the left hand of the school (the student register systems) to know what the right hand (the Capita system) was doing is not dissimilar to the circumstances of the recent court case of Ferguson v British Gas where the defences put forward by British Gas that erroneous debt collection letters were ‘computer generated’ and so they couldn’t have been harassing the plaintiff were dismissed by the Court of Appeal in England and Wales.

So we can add a potential legal risk to the list of reasons why this is an IQTrainwreck.

Parents of dead children asked to choose school for them

1 Reply

According to BBC News Gloucestershire County Council has sent out letters to the parents of a number of children who had died asking them to choose a school as they approach school starting age!

This has resulted in a good deal of pain and grief for the parents and a significant embarrassment to the council and is one more strand in the perception that government agencies (both central and local) have a very bad record in in Information Management. The problem seems to have arisen because of poorly defined inter-agency information requests (i.e. the council seems to have asked the local NHS Trust for details of children born between certain dates and not specified that they should be still alive) and then poor checking of results (at least that’s what the council is blaming it on).

We all know that one can’t add quality so this PR disaster shows the importance of getting the spec right. It also raises questions: “How do the Council find out about children who have moved into the the area after being born within the purview of another NHS Trust?” 

Monkey See… Monkey Do

Leave a reply

Courtesy of Nigel Thomas comes this story of an IQ Trainwreck that eminates originally from the hallowed halls of Harvard University, where monkeys swing freely through the Ivy (original story previously published at www.thedailywtf.com as “I’ve Got the Monkey Now”).

The Harvard Business School Publishing redeveloped and relaunched their website back in 1999. Part of the QA was a series of test packs for their various search scenarios, including a test for ‘single result returned’. Being concientious people, the Harvardians regularly ran tests on the live system using the test packs to verify that the system was running correctly. Their single result test case was expected to return just one result.. for the search term “monkey” it was to return one article called “Who’s got the monkey?”. They put it in the shopping basket and completed a dummy sale to a dummy account (Mr 123, 123 one hundred and twenty third street… that kind of thing)

One day, in 2002, that test failed. An update to the paper “Who’s got the monkey?” had been commissioned and published called “Who’s got the monkey now?”. Why had that been commissioned?

 Well, it seems that the folks in the marketing department were looking at sales trends. How many of what was bought when? That kind of thing. Basic Business Intelligence.

“Hey Mike”, said a fictionalised HBSP marketing analyst, “there’s a lot of monkeys being sold on the site. Maybe we should upgrade the monkey”. “Yes Jim”, said Mike (who we must imagine as being a pipe smoker with a moustache for the necessary air of gravitas to be tangible), “that’s a lot of monkeys. Time to activate Marketing plan Ultra-Overdrive A!” (that conversation probably never happened like that, but in my head it made me laugh, so I thought I’d share it).

They produced summaries of the paper, pamphlets on the paper, knicknacks with monkey themes, widely publicised the success of the paper, and commissioned an update.

Which one day in 2002 wound up appearing in the single result search test cases run by the website team, causing the test to fail.

Why is this a trainwreck?

Well, Harvard spent a lot of money publicising a paper as a best seller that had reached that point most likely through test data being read as ‘valid’ when taken out of context. The ‘dummy’ addresses and dummy accounts could have (and should have) been excluded from the reports being used by the Marketing Analysts.

As an aside… I regularly get snail mail marketing from Harvard Business School addressed to Daragh O. Brien (note – part of my surname has become an initial).  Those of you who read the trainwreck about direct mail and direct email I posted last week can probably guess what happens to those letters.