After the ABS Census computer was shut down on Tuesday night, one bright spark on talk-back radio said we wasn’t going to fill in the census data now. What’s the point of giving information to a government that can’t run a census, he asked. If they can’t do a simple thing like that then they can’t provide all the health, education, police, infrastructure services we need.
On radio we were told this morning that the Census needs 95% of us to supply accurate data for the data to be useful in planning future services. We are told the computer system will be up and working today. They may yet pull it off, but significant confidence has been lost and the ABS has trashed its brand.
While it’s easy to blame the ABS, if we go back to 2014 we find that budget cuts had just about brought the ABS to its knees. It had lost 350 staff, they lacked a chief statistician, essentially a chief executive, and with an recruitment freeze it wasn’t clear when they would get one. Their computer systems were so old some were no longer serviced by the companies that supplied them.
In May 2015 the tide turned, with a $250 million technology upgrade included in the budget. The 2016 census was saved.
- The bureau’s [new] chief David Kalisch said the money would enable the ABS to perform an entire technology upgrade with most of its systems to be replaced with three common platforms; one for data collection, one for processing and one for statistical products.
The new systems would connect “computer to computer” to other parts of government taking in data such as births, deaths and marriages and immigration statistics. Over time the bureau would be able to expand to collect data automatically from organisations such as the Tax Office.
- Mr Kalisch said he was delighted and encouraged by the confidence that the government had shown.
The ABS will now deliver the full census on August 9, 2016 as originally planned with preparations well advanced. It will be Australia’s first “digital first” census, with two-thirds of households expected to complete their forms online.
I believe the ABS saved $100 million by doing the census online.
On radio yesterday I heard from two people who had worked at census time in the past, delivering forms personally, offering help and advising migrants that it was safe to tell the government their religion, and collecting the forms. Obviously an expensive way to go, but excellent PR, with a quality data collection outcome.
In the event the ABS used IBM as an outside contractor to do the job for $9.6 million.
However, the online modality brought out all the concerns about privacy, unfortunately with a galaxy of computer nerds and others, most of whom hadn’t bothered to find out how the ABS was handling the situation, delivering opinions and spreading concerns . ABC radio did a really good job of sharing ignorance and misleading information.
David Glance explains the ABS methodology of a linking key between the names and the information. What he didn’t say was that the information, the names and the keys were to be kept on three different computers all offline, which I understand to be the case.
Obviously there was a period of vulnerability when the information was being entered online. Online entry was an option last Census, but not the frontline method.
The Privacy Commissioner signed off on the process, and was to be involved in supervising what happened. Nothing’s perfect, he says, but good as.
Glance suggests the privacy battle has effectively been lost by the ABS. Seven senators indicated they would not be putting their names on the form, and the Greens indicated they would consider legislating to prevent this happening. Some experts, learned in law, suggested the ABS only had a head of power to collect statistical information, questioning their ability to legally enforce the inclusion of names. We are certain to hear more.
At our place we decided to order a printed form in case there was a computer stuff up, and it duly arrived on Monday. Wise decision.
David Glance and Mike Johnstone are among those who questioned whether the ABS census computer crash was actually caused by a ‘distributed denial of service attack’ or by a simple overload of the system. The system was only tested to take a million forms an hour.
Robert Merkel has done the numbers:
- around 18 million Australians live in the eastern states, which equates to about 7 million households.
If even 50 per cent of those households attempted to submit their census during the evening hours from 7pm to 9pm, that would equate to 1.75 million form submissions per hour, 75 per cent more than the reported capacity of the site.
Furthermore the ABS would need to plan for spikes at any time within the peak period.
No surprise then that the system crashed at 7.30 pm, and was closed down by the ABS at .
ABS insist, however, that they suffered a hostile attack from outside the country. Then a router failed, and then their monitoring picked up information inside the system that wasn’t kosher.
At that point they closed it down, while automatic tweets were still going out to encourage people to log on, and didn’t tell the public what had happened until next morning.
We are assured that no census information was destroyed, altered or stolen.
However, it does seem that capacity planning was poor. Even a child could see that a million forms an hour was not enough and it would have taken very little extra hostile traffic to crash the system. Denial of service attacks are common and to be expected. Contingency planning, if there was any, proved to be inadequate.
- There are a number of ways in which the dangers of a DDoS can be mitigated. It is unknown at this point what measures the ABS and its contractors took to prepare for the possibility.
Turnbull is saying that the ABS did not do all that it should have done to protect the process. We can assume that heads will roll.
A couple of weeks ago Kelly Higgins-Devine on local radio devoted an hour to the census, including its history and what other countries do. It seems that many do not need a census because the routinely collect enough information to do the job. Glance says the rationale for collecting and keeping our names was simple:
- With names and addresses, the Census data can be linked to other data sets where we have already allowed our name and address to be used. This includes health, education and other data. Together, they should help give a more complete and accurate picture of how the distribution of people in Australia matches present and future services.
From a population health perspective, linked health data may also reveal underlying health trends or relationships between age and income, or general health outcomes, that were not possible to see without this linkage.
Are we worrying unnecessarily?
On IT, though, we do not seem well-placed to participate in the information economy. As I went to publish this post this morning my interconnection dropped out. A few hours later it has decided to work.
James Turner in the AFR says the census crash was a disaster waiting to happen. There was a large gap between assurances the ABS gave over recent weeks and what actually happened. They are going to have to work hard to win back our trust and it is critical that they do.