Electronic health record system crash

Thishas been inconvenient.

The “network outage” began around 10:30 p.m. Monday and affected all 24 hospitals in the Sutter system. It was caused by the activation of the fire- suppression system in Sutter Health’s data center in Sacramento, according to an internal message sent by text to Sutter employees.

They call it a "Code Lime"  Code Red is fire, they have a kaleidoscope of colors.  They canceled all the elective surgeries, did everything on paper.  Worst inconvenience has been the inability to access records.   Was still out last night.  

I specifically asked them about this very thing happening prior to purchasing the software.  What if you have a fire or something at your data hub and it all crashes?  "That can never happen, there are redundant systems and backups."  Have not checked to see if it is on line yet today.

Yeah.  "Have you ever tested it?"  "Sure we did..."

There was no good reason for this to happen to something so important and expensive these days. There should be a "cloud" backup server to fail over to. The little 50-person 3-office operation where I work has one for email and we're working on one for VoIP (right now we just have a backup VoIP server in the same room, with data backed up offsite).

Edit: And we're not even doing anything remotely important.

The primary system is cloud based, so it seems like a back up would not be that rough to have in place.  That is how those MFers told me it was set up.

Now mine is up, but my office manager can't log in, doh

In reply to GameboyRMH :

You act like the proper risk assessment was done when these types of systems are designed or emplaced. Most of the time money is just thrown at it because having the most expensive system strokes the ego of the C-Suite. The Project managers always talk a big game but most don't have the technical expertise to make that assessment. I'd like to say that isn't the case but I sell to decision makers on a regular basis. Unless they're personally involved in the day to day activities of the organization, berkeley the people that need to use it.

Hopefully your admins get it cleared up quick Toebra.

Toebra said:

The primary system is cloud based, so it seems like a back up would not be that rough to have in place.  That is how those MFers told me it was set up.

Huh, if the primary system is cloud-based, then it's super duper easy to have a backup system in place.

In reply to GameboyRMH :

I'm assuming it's a "Private Cloud" rather than a "managed cloud" installation like AWS, Azure, or Google. Something probably severed the datalink to the datacenter so the "cloud' portion is out.

Marketing people are fun.

Haaaaa.

I used to work with that system in that office. 

Private cloud is probably the best way to describe it (though it has been a while). Can't believe they didn't have an instant failover ready though. You should have had read-only  access 100%. If you think it's bad at Sutter, the folks at the emr vendor who let this happen are probably much worse off right now.

I did most of my work on the hospital emr side in the central valley region, so I spent a lot of time in Modesto too.

Oooof.

GameboyRMH said:

Huh, if the primary system is cloud-based, then it's super duper easy to have a backup system in place.

I know, right.  Looks like some communication node got wiped out and the entire house of cards fell over.  

I am still waiting to see this whole thing save me a little bit of time, instead of eating up the day.