Daily numbers, graphs, analysis thread 21

[BigChocFrenzy]

Welcome to thread 21 of the daily updates

Resource links:

Uk dashboard deaths, cases, hospitals, tests - 4 nations, English regions & LAs
R estimates UK & English regions
Imperial UK weekly LAs, cases / 100k, table, map, hotspots
UK School statistics Attendance
Modelling real number of UK infections February to date
NHS England Hospital activity
MSAO Map of English cases
Cases Tracker England Local Government
ONS MSAO Map English deaths
CovidMessenger live update by council district in England
Scot gov Daily data
Scotland TravellingTabby LAs, care homes, hospitals, tests, t&t
PH Wales LAs, tests, ONS deaths
NI Dashboard
Zoe Uk data
UK govt pressers Slides & data
ICNRC Intensive Care National Audit & Research reports
NHS t&t England & UK testing Weekly stats
PHE Surveillance reports & LA Local Watchlist Maps by LSOA
ONS England infection surveillance report each Friday
Datasets for ONS surveillance reports
ONS Roundup deaths, infections & economic reports
ECDC rolling 14-day incidence EEA & UK
Worldometer UK page
Our World in Data GB test positivity etc, DIY country graphs
FT DIY graphs compare deaths, cases, raw / million pop
Alama Personal COVID risk assessment
Local Mobility Reports for countries
UK Highstreet Tracker for cities & large towns Footfall, spend index, workers, visitors, economic recovery

⏭ Our STUDIES Corner ⏮

We welcome factual, data driven and analytical contributions
Please try to keep discussion focused on these
📈 📉 📊 👍

UK is currently at alert level 4? I'm wondering if we will go up to level 5 over the next few months?

Or would that be vetoed as so economically disasterous? Plus the population is becoming very peed off with the Government and it's "stay home, go to work, don't go to work" announcements.

Followup on medical software guidelines

1. There is a new guideline out (2019)

file:///C:/Users/Manci/Desktop/mdcg201911guidancequalificationclassificationsoftware.pdf It basically tells you when something is considered to be under regulation or not. Unfortunately medical software that does not handle patient data, or has a role in diagnosis or treatment can be exempt. In very broad terms.

1. According to this, the gov's dashboard is more of a CIS (Clinical information system) and is actually not a medical software. Fair enough potentially. And it is more classed as CDM - Clinical Data Management software.

1. it seems -from 10,000 feet view- that it is all too easy to bypass these regulations. As soon as you take out the medical profession of it (eg: doctors don't need to use this in their work with the patient) then it becomes a data regulatory issue.

1. Data reg is very strict in the EU so I would be interested in how they are qualifying those criteria. But let's be fair: they don't and the EU doesn't care any more and parliament is too stupid to even ask these questions. That's why cummings et al can get away with shoddy software and data manipulation.

@RedToothBrush
Agreed on many points. But I think your husband might be slightly naive or me too cynical. I do think it is more likely that they are transferring all data into a central database. Think about it: local dbs have some form of local control and validation and to access data you actually have to go through people. MEssy and time consuming.
If they make it mandatory to send full datasets per person* than all those messy and time consuming rounds with local databases can be spared at the moment. Any time in the future they need to know anything they can mine their own full dataset that is tucked away some place similar to cambridge analytica datasets....

*and at point of conception time was so tight that nobody questioned this decision, or just earmarked it as stupid

[quote TheSunIsStillShining]@RedToothBrush
Agreed on many points. But I think your husband might be slightly naive or me too cynical. I do think it is more likely that they are transferring all data into a central database. Think about it: local dbs have some form of local control and validation and to access data you actually have to go through people. MEssy and time consuming.
If they make it mandatory to send full datasets per person* than all those messy and time consuming rounds with local databases can be spared at the moment. Any time in the future they need to know anything they can mine their own full dataset that is tucked away some place similar to cambridge analytica datasets....

*and at point of conception time was so tight that nobody questioned this decision, or just earmarked it as stupid
[/quote]
Well he did say that it could be that they are transferring entire nhs files of people. But he was trying to think of alternative explanations to give the benefit of the doubt.

I personally will not engage with track and trace on the basis of how much data is being passed into private hands through it. Its one reason I really am limiting my movements. And this incident only reinforces that feeling tbh. So its not exactly good for public trust...

One comment i do have - surely they teams monitoring their disk space?

I know they can fill up very quickly, but it should have been spec'd to deal with hundred of thousands of tests a day.

Although, probably their testing would only have involved very limited data!

@Timeforanotherusername

One comment i do have - surely they teams monitoring their disk space?

I know they can fill up very quickly, but it should have been spec'd to deal with hundred of thousands of tests a day.

Although, probably their testing would only have involved very limited data!

Teams?

I think you are over estimating how many people are involved in the project and what they are responsible for.

I strongly suspect there won't be someone with an exclusive job of monitoring disk space. Never mind a team.

^
I strongly suspect there won't be someone with an exclusive job of monitoring disk space. Never mind a team.^

Well that would be very worrying indeed! As most large organisations would be monitoring this type of thing.

Thanks @littleowl1 for the info. You're ahead of the bbc at the mo for rate per 100,000!

Jeeze the top council lists!

@Timeforanotherusername

One comment i do have - surely they teams monitoring their disk space?

I know they can fill up very quickly, but it should have been spec'd to deal with hundred of thousands of tests a day.

Although, probably their testing would only have involved very limited data!

If it's azure (or any cloud) they can set to spin up new storage space automatically if x threshold is met. Email sent later FYI. If money is no issue this is what I would do.

just flagging that we are closing in on 1000 posts...

Good point Born. I was just thinking there's no halls or huge mixing like there is at uni. I went to two of the Stockport colleges and it was so different to uni socially. What baffles me most is how the previously shielded have been forgotten in all this.

Official line seems to be:

"In his London Playbook briefing for Politico Europe, Alex Wickham has some detail about what actually went wrong.

WHAT WENT WRONG: The problem occurred when test result data from labs wasn’t successfully transferred onto the actual dashboards that report the numbers. PHE says some files containing positive test results — unbelievably — exceeded the maximum file size that can be loaded onto their central system, and so they were missed. It’s now splitting the large files into two so it doesn’t happen again. Seriously."
ffs

I personally will not engage with track and trace on the basis of how much data is being passed into private hands through it

I assume you engage with healthcare though? No difference. Not engaging with track and trace is not helpful to you or anyone else.

Sun

What must they have been transferring?

It is actually quite believable.

What they should have in place though is some checks which would highlight that not all files have been loaded.

Although it sounds as if they disappeared into a black hole.

I almost wonder if they 'fixed' a bug around 24th Sept and introduced a bigger one.

What you say about patching patches etc is so true.

And its often just a quick fix that is put in place.

So after all the reports of cases slowing and even starting to become stable it seems it wasnt and hasn't?
Is our R rate higher now do you think?

And the past couple of days our cases were around 12000 and the week before all around 10000 (now the added tests are now added to the dates?).
Are all the extra positive cases now included?
Sorry just trying to catch up and get my head around this total fuck up.

The ZOE app, which shouldn't have been affected by this shit-show, said that they estimated the R rate had decreased on average across the country.

I have no inside knowledge on this but I dont think we should assume local people have been getting the wrong data. Quite a few places seem to have stopped relying on the national system and turned to local sources. This might explain why, say, Manchester and London mayor's have been kicking off for a while.

The data problems affected the whole country but were obviously more serious for areas that were already out of control. The north west really needs very stringent measures, the east, south east and south west still dont. We do need the economy functioning anywhere where it is possible to do so without overwhelming health services.

We've never been identifying and isolating all the infected, that has not changed, only the gap between infected and traced.

The Scottish MP can, and should, be referred to the standards committee. If they suspend her from Parliament for 10 days her constituents then get the opportunity for a recall petition. As she put MPs at risk I imagine parliament would be happy to suspend her.

apologies for the daily fail quote but if this really is the problem then it's quite staggeringly basic

"The problem was caused by an Excel spreadsheet reaching its maximum file size, which stopped new names being added in an automated process.
The files have now been split into smaller multiple files to prevent the issue happening again"

"I don't know if things have changed, but a couple of weeks ago the media were reporting on an increase in women aged 20-40 being hospitalised for Covid."

@SheepandCow Yes, they were. Unfortunately I can't find any publicly available source of data on hospitalisations by age and gender. Strangely they do not seem to be sharing this with us. It would be of great interest to me and others I am sure!

I would really like to see proper analysis if hospital admissions currently.

Utter Anecdata but a friend who's relative works at a small local ish hospital said their Covid beds were all full (NE.)

Ed Conway of Sky has a good twitter thread about the whole shambles.

"The problem was caused by an Excel spreadsheet reaching its maximum file size, which stopped new names being added in an automated process.
The files have now been split into smaller multiple files to prevent the issue happening again"

Err... does that mean that the pre-processing of data is happening using Excel macros? I mean, seriously, you could do something better using awk.

They're using excel? Really?

Thought I'd have a browse on the civil service jobs site and found this. The job spec sounds a bit confused to me, surely you want top notch data scientists running the data side but then surely public health experts interpret that and set alert levels? (I know it's a collaboration but it sounds here like they want someone to do both).

Anyway, would be interested in what the data experts here think of this job ad (for 11 positions) and if any of you wanted to apply... deadline extended to 14th October.... work from home.....

On the other hand bloody depressing they are recruiting in this timeframe. They need people able to do this NOW. We're fucked aren't we?

www.civilservicejobs.service.gov.uk/csr/index.cgi?SID=cGFnZWNsYXNzPUpvYnMmcGFnZWFjdGlvbj12aWV3dmFjYnlqb2JsaXN0JnNlYXJjaF9zbGljZV9jdXJyZW50PTQmam9ibGlzdF92aWV3X3ZhYz0xNjg2ODM1Jm93bmVydHlwZT1mYWlyJm93bmVyPTUwNzAwMDAmY3NvdXJjZT1jc3FzZWFyY2gmdXNlcnNlYXJjaGNvbnRleHQ9MTEwNDE4MzY3JnJlcXNpZz0xNjAxODkwNDM2LThmYTVkYjY4Njg5MzllMjBhNzU2MDdmY2JkYTQyNmE4NTY0NzU2Nzg=

Can someone explain the failure using language that a primary school child could understand. I've followed for ages and get most of it but I don't understand about files getting too big, where does the data stay, why doesn't someone notice, what is the data, is it a yes Ms Crock is positive or something else?

Wave

It is sounding now as if the Excel spreadsheet ran out of rows.