<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=138560526769886&amp;ev=PageView&amp;noscript=1">

4 Data Loss Horror Stories to Learn From

The Soup


Businessman leaning away in fear of shadow monster

Horror films and ghost stories are the cause of nightmares for many, but when it comes to CIOs, CTOs, and IT managers, there's a different monster hiding in the closet. It's more terrifying than any ghoul or creepy clown, and those who've experienced it know they never want to go through it again.

Its name? Data loss.

Cue eerie music.

If those two words made you flinch, you'll probably be able to empathize with the tales we're about to tell. In this blog post, we share four data loss horror stories where human error, water, construction, and power loss are the cause of corporate nightmares.
 

Man covering his face with both hands in anguish


1. Human Error

"We accidentally deleted production data and might have to restore from backup" is the Tweet that a Git repository hosting service known as GitLab sent out on January 31st of this year. The statement is sure to send shivers down any sysadmin's spine.

On their blog, they explain that they experienced a "major service outage" for one of their products when a GitLab engineer accidentally removed data from their primary database server, causing their service to be unavailable for many hours.

They lost production data that they were ultimately unable to recover, estimating that around 5,000 projects, 5,000 comments, and 700 new user accounts were affected by the loss.

Tweet from GitLab explaining accidentally deleted production data
We can only imagine the stress that their team faced in that situation, as many sysadmins and developers reached out in sympathy in response to the sobering situation. It is a very real and important reminder that human error happens, and evaluating your data recovery plan is as important as ever.

Human error is a prominent cause of data loss and includes:

  • Accidentally formatting a hard drive
  • Unknowingly deleting a file or folder
  • Physically damaging the computer system

Let's face it: we're all human. Even if you're the most knowledgeable and talented techie in the building, you're still one swift click away from accidentally losing everything.

Hard drive splashing and sinking down into a pool of clear water


2. Water and Other Liquid Damage

"Rick, Navy Boulevard is flooded. The water is heading right towards your office."

These were the gut-wrenching words that Independent News publisher Rick Outzen heard his landlord shout over the phone to him on the night of a torrential downpour. By the time Outzen had reached the office, water was up to his knees and filled the building.

With help, Outzen "unplugged computers and yanked the waterlogged hard drives off the floor" while papers floated everywhere around him.

Their staff lost "valuable time" because of the flood water, and while Outzen didn't specify any data loss, it can be inferred that the waterlogged hard drives he'd found meant exactly that. Outzen stated that the flood proved as bad a disaster for their weekly paper as Hurricane Ivan in 2004.

While water damage can include natural disasters like hurricanes, floods, and storms, water damage isn't limited to mother nature striking your office. Leaks from the water heater or air conditioning can also be the culprits behind damage to your data center.

Data loss could even result from something as simple as coffee accidentally spilt by an employee onto a computer. It sounds like a cartoon moment, but it does happen, and that type of spill can make for quick and irreversible data loss.

Hooded man in a dark forest standing behind a shovel dug into the dirt


3. Construction Work Accidentally Cutting Fiber Cables

Imagine that you're working on a critical project for a client and then suddenly—bam—the internet goes out. You check outside your office and sure enough, a construction crew is digging around the corner. You realize they must have accidentally severed a cable while working.

Not only are you then forced to waste valuable time before the cable is fixed, but when you finally get internet back to finish the project you were working on, panic sets in. You realize you lost hours of work on the project that wasn't backed up properly when the outage happened.

As our cities and towns grow, fiber cuts are becoming more common—you can see it often in the news. Everything from hand shovels and backhoes to post-hole augers can stop network traffic in its tracks by severing your fiber optic cable. And if your fiber connectivity gets cut at the wrong time? Goodbye, data.

Lightning striking across a dark, cloudy sky with a power line and leafless tree in the foreground


4. Power Loss

The National Security Agency’s Utah Data Center, a $1.2 billion complex covering 1.5 million square feet, had to delay its opening due to electrical problems. With 10 "meltdowns" and "power surges" in the past 13 months, it meant that "chronic electrical surges ... have destroyed hundreds of thousands of dollars' worth of machinery and delayed the center's opening for a year."

A power outage or power surge can mean that your hard drive, server, or other infrastructure suddenly shuts down, which can damage—or even destroy—the infrastructure if this happens repeatedly.

Power loss has a variety of causes, including:

  • Equipment failure (due to age, performance, etc.)
  • Wildlife and tree interference
  • Weather (lightning, high wind, etc.)
  • Car crash or construction equipment damaging lines

One noteworthy part of the story is that the Utah Data Center “requires 65 megawatts of electricity and its own power substation,” meaning it generates an enormous amount of heat.

It’s an often overlooked aspect of power outages, but without proper cooling, temperatures in the server room can go from an acceptable range of 72-80 degrees to well over 100 degrees Fahrenheit in a matter of minutes. This causes servers and other infrastructure to shut down. And without servers being shut down in a controlled manner, there will be data loss and potentially unrecoverable and/or corrupt databases.

Your network, servers, and applications need to be protected by redundancies in power infrastructure. Putting your assets in a datacenter offers the best chance of survivability from the horrors that keep you up at night so that you can enjoy sweet dreams.

The word "fear" in all caps and crossed out with an X written like chalk on a chalkboard

 

Don't Be the Next Data Loss Horror Story—Safeguard Your Data with LightBound!

 

 

Don't wait for disaster to strike—ensure your data is safe and minimize downtime with LightBound's Cloud portfolio. We offer solutions for superior disaster recovery and redundancy, including:

With LightBound, you can ensure continued operations even in the face of power loss, construction mishaps, flooding, and even accidentally deleted databases. Contact LightBound today to get started and banish those IT nightmares for good!

Learn More

October 19, 2017 / by Dane Dittemore