After the ABS Census computer was shut down on Tuesday night, one bright spark on talk-back radio said we wasn’t going to fill in the census data now. What’s the point of giving information to a government that can’t run a census, he asked. If they can’t do a simple thing like that then they can’t provide all the health, education, police, infrastructure services we need.
On radio we were told this morning that the Census needs 95% of us to supply accurate data for the data to be useful in planning future services. We are told the computer system will be up and working today. They may yet pull it off, but significant confidence has been lost and the ABS has trashed its brand.
While it’s easy to blame the ABS, if we go back to 2014 we find that budget cuts had just about brought the ABS to its knees. It had lost 350 staff, they lacked a chief statistician, essentially a chief executive, and with an recruitment freeze it wasn’t clear when they would get one. Their computer systems were so old some were no longer serviced by the companies that supplied them.
In May 2015 the tide turned, with a $250 million technology upgrade included in the budget. The 2016 census was saved.
- The bureau’s [new] chief David Kalisch said the money would enable the ABS to perform an entire technology upgrade with most of its systems to be replaced with three common platforms; one for data collection, one for processing and one for statistical products.
The new systems would connect “computer to computer” to other parts of government taking in data such as births, deaths and marriages and immigration statistics. Over time the bureau would be able to expand to collect data automatically from organisations such as the Tax Office.
- Mr Kalisch said he was delighted and encouraged by the confidence that the government had shown.
The ABS will now deliver the full census on August 9, 2016 as originally planned with preparations well advanced. It will be Australia’s first “digital first” census, with two-thirds of households expected to complete their forms online.
I believe the ABS saved $100 million by doing the census online.
On radio yesterday I heard from two people who had worked at census time in the past, delivering forms personally, offering help and advising migrants that it was safe to tell the government their religion, and collecting the forms. Obviously an expensive way to go, but excellent PR, with a quality data collection outcome.
In the event the ABS used IBM as an outside contractor to do the job for $9.6 million.
However, the online modality brought out all the concerns about privacy, unfortunately with a galaxy of computer nerds and others, most of whom hadn’t bothered to find out how the ABS was handling the situation, delivering opinions and spreading concerns . ABC radio did a really good job of sharing ignorance and misleading information.
David Glance explains the ABS methodology of a linking key between the names and the information. What he didn’t say was that the information, the names and the keys were to be kept on three different computers all offline, which I understand to be the case.
Obviously there was a period of vulnerability when the information was being entered online. Online entry was an option last Census, but not the frontline method.
The Privacy Commissioner signed off on the process, and was to be involved in supervising what happened. Nothing’s perfect, he says, but good as.
Glance suggests the privacy battle has effectively been lost by the ABS. Seven senators indicated they would not be putting their names on the form, and the Greens indicated they would consider legislating to prevent this happening. Some experts, learned in law, suggested the ABS only had a head of power to collect statistical information, questioning their ability to legally enforce the inclusion of names. We are certain to hear more.
At our place we decided to order a printed form in case there was a computer stuff up, and it duly arrived on Monday. Wise decision.
David Glance and Mike Johnstone are among those who questioned whether the ABS census computer crash was actually caused by a ‘distributed denial of service attack’ or by a simple overload of the system. The system was only tested to take a million forms an hour.
Robert Merkel has done the numbers:
- around 18 million Australians live in the eastern states, which equates to about 7 million households.
If even 50 per cent of those households attempted to submit their census during the evening hours from 7pm to 9pm, that would equate to 1.75 million form submissions per hour, 75 per cent more than the reported capacity of the site.
Furthermore the ABS would need to plan for spikes at any time within the peak period.
No surprise then that the system crashed at 7.30 pm, and was closed down by the ABS at .
ABS insist, however, that they suffered a hostile attack from outside the country. Then a router failed, and then their monitoring picked up information inside the system that wasn’t kosher.
At that point they closed it down, while automatic tweets were still going out to encourage people to log on, and didn’t tell the public what had happened until next morning.
We are assured that no census information was destroyed, altered or stolen.
However, it does seem that capacity planning was poor. Even a child could see that a million forms an hour was not enough and it would have taken very little extra hostile traffic to crash the system. Denial of service attacks are common and to be expected. Contingency planning, if there was any, proved to be inadequate.
- There are a number of ways in which the dangers of a DDoS can be mitigated. It is unknown at this point what measures the ABS and its contractors took to prepare for the possibility.
Turnbull is saying that the ABS did not do all that it should have done to protect the process. We can assume that heads will roll.
A couple of weeks ago Kelly Higgins-Devine on local radio devoted an hour to the census, including its history and what other countries do. It seems that many do not need a census because the routinely collect enough information to do the job. Glance says the rationale for collecting and keeping our names was simple:
- With names and addresses, the Census data can be linked to other data sets where we have already allowed our name and address to be used. This includes health, education and other data. Together, they should help give a more complete and accurate picture of how the distribution of people in Australia matches present and future services.
From a population health perspective, linked health data may also reveal underlying health trends or relationships between age and income, or general health outcomes, that were not possible to see without this linkage.
Are we worrying unnecessarily?
On IT, though, we do not seem well-placed to participate in the information economy. As I went to publish this post this morning my interconnection dropped out. A few hours later it has decided to work.
James Turner in the AFR says the census crash was a disaster waiting to happen. There was a large gap between assurances the ABS gave over recent weeks and what actually happened. They are going to have to work hard to win back our trust and it is critical that they do.
12 thoughts on “Census crash”
Further to Robert Merkel’s numbers, according to this article on July 20, Census Australia tweeted
The ABS expected no more than 500,000 lodgements each hour (with peaks of 1 million) – a guaranteed fail. If we really had a Westminster system of ministerial responsibility, McCormack would have fallen on his sword yesterday.
Who decided to use IBM ?
Surely the QLD Health Payroll disaster would put a scratch through them straight up.
Cobio I think….
A million an hour converts to 277 per second. Turnbull said that lodgements peaked at 150 per minute.
However, it was almost certainly going to run out of capacity at some time during the evening.
What’s been stressed is that three things that happened together, the flood of increased traffic, the failure of the router, and noticing some suspicious data inside the system.
What a b*lls-up!
It has to go down with
* leaving Tassie off the Bicentennial logo
* collapse of the Westgate Bridge
* stuffing Joern Utzon around
in the rich annals of ludicrous events we parade before the world. It makes a nice accompaniment to Channel 7’s national chauvinism and unsubtle hounding of world-class Australian athletes who don’t live up to Channel 7’s expectations.
Richly comic is that our PM has advocated new, disruptive technologies; urged us to be agile and innovative; proclaimed that it is our patriotic duty to expect start-ups to fail. Hot air, high hopes. Optimistic vacuity?? And yet info tech can do some basics really well.
Hoist with his own petard, Mr Turnbull apparently brightly tweeted his census completion just minutes before the ABS closed down the carnival of modernity for 43 hours.
You couldn’t get funding for a script as silly as this.
Acclaim for Utopia, its accurate and pungent observation is (so I’m told) viewed with horror and delight by many public servants.
And well done, Robert Merkel. When a back-of-the-envelope estimate shows a flaw in a consulting team’s efforts, you have to wonder what might have been achieved by a really smart team, on a lower budget??? Or by a sceptical department who had employed that consulting group.
The latest report I’ve heard: the PM indicated an enquiry is underway. Its report will lead him to decide who will be sacked (not whether anyone might lose their job).
If I recall correctly, the five year census interval was to be extended to ten years. I can’t recall the timing but it was certainly during the previous government. If the ten year scenario was a real possibility, perhaps that formed part of the antecedents for this cock-up. That might have led to the existing system becoming rusty and unprepared for the current event.
Not making excuses here but there may have been other factors involved.
Geoff, Laura Tingle has the story – neglect and contempt from the Government, some restitution with Project Arthur to renew technology and get the show on the road again. It involved a massive Australian Population Survey of 2.5 million to replace the 2016 census and extend the census cycle to 10 years.
It’s not clear, she says, how and why the 2016 census was reinstated.
Also the chief statistician position was left vacant for nearly a year to 2014. How much happened under Turnbull’s watch as responsible minister is not clear, but I believe there have been three ministers in recent times.
There’s little doubt that they wouldn’t have attempted it unless they could do it el cheapo online.
There is plenty of blame to go around, from IBM, right up to Turnbull himself.
It always struck me when my wife was complaining TAFE paperwork that most of it did not need to be done by every teacher every day because the information was being used for putting together statistics that only needed to be accurate enough for decision making and general reporting.
Much of the paper work load could have been reduced dramatically if basic statistical theory had been used to work out the frequency of reporting required to provide the required level of accuracy.
I suspect that the same applies to most, if not all of the information collected on census day. Every household doesn’t have to fill in a census form and not every household has to answer every question. In addition, some of the questions could be answered by using the statistical data held by people like the taxation dept etc.
I realize that Joseph and Mary had to go to Bethlehem because of the Roman census and the romance of taking a nationwide snapshot on a particular day. However, the work I have done for sample size calculations suggest that appropriate stats could be collected for most questions using much much smaller samples than the one we are using.
“What have the Romans ever done for us?”
Let’s have a look….
Till the last election, then the new bloke.
Turnbull was never ” Minister Responsible ”
Not defending the tosser, just correcting a misunderstanding.
Jumpy, thanks for sorting out ‘responsible minister’.
John, some sort of competent inquiry is needed to have a whole look at what the ABS should be doing and then provide the resources to do it. That comes back to the “government”. We will certainly have a senate inquiry, but one of the things I learnt from the senate climate change inquiry is that the Senate is not the Government.
I could be wrong, but I think the ABS has a legal obligation to do a census.
Elsewhere John Quiggin has a post in which he says he can’t see how the exercise can be saved.
He links to an article by Peter Martin which, inter alia, tells us they want to keep our names so they can conveniently link it with PBS and Medicare information, and then sell info, presumably to Big Pharma to help them with their advertising. The new bloke apparently a man with an entrepreneurial spirit that would please Malcolm.
Then he links with Chris Graham at New Matilda, where we learn that the contract with IBM was written in October 2014, which would place it before the new bloke turned up at ABS and when Steve Ciobo was the responsible minister.
IBM were the mob who contracted to deliver the Health payroll system to the Qld Govt for $6.19 million. It ended up costing us $1.18 billion, and we got nothing back from IBM.
Graham also tells us that the decision to keep our names was cynically announced just before Christmas.
He also points out that the ABS information was sent “To the Householder”, which many chuck straight into the bin.
The workers at ABS are “gutted” over what happened to the census according to Nadine Flood of the CPSU.
Apparently 700 staff have been lost since the last census was done.
There’s a thing called the ‘productivity dividend’, also known as the salami slice method of downsizing. Every year at budget time you shave the entity’s budget and wait to see whether it falls over. When it does you blame it, or say the service is so bad we may as well do without it.
It seems they have been taking more than salami slices out of the ABS.
Comments are closed.