Thursday, February 2, 2017

When the Data Goes Down the Drain, It Pours

It seems like paying attention to a particular topic tends to surface many previously unremarked instances of the same thing. So today I also ran across this blog post, Saved by the backup – Do you have a backup plan in place?, by my friend and fellow Access MVP, Anders Ebro Christiansen, also known as the Smiley Coder in online forums.

If you don't have a good backup plan in place, you better have a good exit plan ready.

Just saying.


Back Me Up, Rambo, I'm Going In....


So, the day started with a phone call from a friend and former colleague. Did I have time to go with him to his client’s site to help restore his Access/SQL Server application following a Ransomware attack?

That sounded interesting, so I went along. After all, how hard could it be? A PITA, as it turned out.

We got there. The owner's face had that hollow look which said he’d lost most of his business data—and he has.

 The rest of the story.

My friend migrated the client’s Access mdb to SQL Server Express on a brand-new dedicated computer over a couple of months last summer and fall. Being on a workgroup, not a domain, my friend had asked me to help with the connections back then. We got it all working. The time frame was late October. The last copy of their data was a “development” .mdf my friend used to fine tune the Access FE for deployment from that date.

Guess what, that is also the most recent copy of the data left in the world, and it’s partially made up of sample data my friend used while working on it. (I “think” that is the case, at any rate. Some of the data I saw wouldn’t make sense otherwise, but, under the circumstances, I really didn’t get a good look at it, as you can imagine.)

The client also had a pre-migration mdb with his business data from April of last year.

What happened? A ransomware attack. Somehow the hackers got control of the new box where the BE SQL Server instance was located. All of the working files on it were converted to “.wallet” files, encrypted of course. Well, we didn't see them, so we assume all, or most of them, were encrypted.

The client must have panicked, because he formatted that hard drive and only then called my friend. Oops. It turns out that all of his daily backups of the master .accdb Front End, as well as the .mdf  that contained the business data itself--i.e. the Back End--were right there on that same HD.

ALL OF THEM.

And he had already formatted that HD before he contacted my friend for help.
 
Fortunately for my friend's peace of mind, he had obtained and installed third-party backup software for the SS Express instance, when he turned it over to the client but, unfortunately, the client had set it up to do the backups only on the same drive ... and nowhere else. 

Why the ransomware had not spread to the rest of his network I don’t know. My only previous exposure to such an attack was that it began to spread across the entire network, picking off Office application files as it found them. This one seems not to have gotten off the source computer. Maybe it was because he’s on a Work Group. Maybe the rest of the office computers were turned off at the time--this happened on Saturday afternoon.  In any event, he indicated he had stopped the intrusion at the original source computer. I am not 100% convinced of that, by the way. Let's just say I think it would be unusual for a ransomware invader to honor a network boundary like that.

All in all, though, that business owner is screwed.
  • The most recent available copy of his business data may or may not be clean.
  • That copy of the data is at least three months old, maybe more.
  • It does include at least 6 months more data than his only other copy of the data, so they’ll probably go with it.
Someone in his office has the unenviable task of sitting down with paper files and rekeying months of orders. By my guess several thousand of them. And then there’s identifying the test data and eliminating that….

All because the only backup was on the same HD as the source. And because he panicked and formatted that HD as his first recovery step (well, maybe he disconnected it from their network first; I don’t know what happened or when).

He told me yesterday he had just purchased and connected external Hard Drives for each of his office computers and was in the process of setting up daily backups for them.

Good first step.

I suggested he add a cloud backup plan as well. I think he will.

When you are talking to your clients, do you offer to include a viable backup protocol as part of your service? Do you ask them to sign off when they decide not to accept your backup plan? Do they understand the risks they are assuming if they don't?