Don’t Fail at Backups
According to BrevAll, only 50% of companies backup 60% of their data. This means that when disaster strikes, they will lose 40% of their data because of their poor backup procedures. So, ask yourself, which 40% is OK to lose?1 If the answer is “we can’t afford to lose 40% of data” then you’ve come to the right place. I have consulted with multiple companies post ransomware attacks where their backups were toast. They only had one copy on the local network and those were destroyed by the threat actor. I have spoken with others who kept multiple backup copies in different places and were OK. Who do you think became fully operational quicker?
Everyone knows they should do backups, and if you don’t, I’m here to tell you that you should do backups on a regular basis in several ways and then practice restoring from those backups at least quarterly. The IT departments have several tasks that they are responsible for and usually backups are one of them. Backups are typically set up once, and then never checked again. Many make the mistake of assuming that backup procedures are running smoothly and that if anything happens, we can just pull from the backup, and everything will be fine. This is a huge mistake. There are several factors that play into a proper backup policy: backup procedures, what needs to be backed up, how often to test backups, how long should the data be kept and why, what types of backups are required, what industry or government regulations oversee the backup policies and procedures.
What are your crown jewels worth?
First, let’s go over the reason for backing up data, what requirements are there, and what kind of data retention policies are there. Backups can be required for certain industries by regulations and define how long they must be retained. For example, the HIPAA Security Rule requires that any written communications regarding actions, activities, or assessments be retained for 6 years as of their creation date or last effective date, whichever is later.2 Even if your industry does not require or regulate backups, it is still considered best practice. Backups ensure that when hardware fails, the data can be restored. Some of the requirements can be hardware, physical and logical space requirements or limitations, and in some cases power and bandwidth. The first step in understanding what requirements there are is understanding what in the organization is to be backed up and what space is needed to host that for however long is needed. Once that is addressed, the second is schedule. How often does this need to be backed up and where are the new backups stored? Does it overwrite the old ones, is there a data retention policy enforcing old backups be kept for x number of months/years? Finally, where is the data physically residing: the cloud, local storage, offsite office or data center? Let’s try and answer some of these questions and in the process ask new questions.
What needs to be backed up? What systems, hosts, servers, etc build your product? What, within the organization, makes the company the most money? These are the crown jewels and should be at the highest priority level of backup. The operating systems of the computers, the data within those computers, the network configurations to make it all talk should all be backed up and ready for restore in the event disaster strikes. Next, who within the organization is critical to keep the business running and what applications and data do they need to function and do their job? Those systems, applications, and resources should be next on the priority list of what to backup. Finally, it’s everything else. The operating systems of all the servers, the network configurations of all the switches, routers, Wi-Fi routers, the remaining data on the network or on shared drives. Many people forget about their network configurations (this includes firewall configurations) and fail to back those up. But it is imperative that those are not forgotten in the event of a disaster.
How should all of this be backed up? It depends. It depends on network throughput, amount of data being backed up, data retention policies and industry regulations or compliance efforts. There are three types of backups that should all be utilized properly. The types of backups are full, incremental, and differential. A full backup is just that. It backs up everything on that system, every time. This takes a large amount of time and is resource intensive. An incremental backup backs up the data that has been changed or altered since the last backup, either full or differential. This requires less time and resources as it backs up less data for each individual occurrence. A differential backup backs up any changes since the last full backup. This is slightly more data than the incremental but less than full. So, a proper answer to the question, is to use all 3 in a proper schedule to be best prepared to restore data when disaster strikes quickly and efficiently without hurting business performance.
To put restoration into context regarding backup strategies, let’s look at a scenario. In this scenario, we do full backups on the 1st of the month, incremental backups each night and a differential backup on the 15th of each month. To restore the system to the 5th of the month. The team would need the full backup from the 1st, and each incremental backup to the night of the 4th. To restore to the 18th of the month, you would need the full backup from the 1st, the differential from the 15th and then each incremental backup from the 15th on. This is best case scenario. Depending on the data size to restore this could take minutes to hours to days to restore this system.
Backup Best Practices
Offsite versus local copies. Local copies provide quick and easy access to the restore points and to restore the data. These local copies provide the quickest way to restore data. However, there is some risk with having local copies. Local copies are subject to the same power outages, floods, ransomware, etc. as everything else on that local network is. This means that if ransomware sweeps through your network, it will likely hit the local backups as well leaving you helpless. Offsite storage assists with that by providing a location away from any physical risks that might affect your local copies. Depending on network setup, these offsite storage solutions can be harder for threat actors to find or hit with ransomware. The key takeaway is to have multiple copies in multiple places with increasing difficulty of access on purpose.
Testing. I have advised countless organizations on how to restore their backups because they had never done it before. Testing is crucial for several reasons. Testing ensures the backups are running successfully and as planned. Next, it trains the IT department in how to work the software and develop processes and procedures on how to do it for others. This practice gives them the muscle memory to be able to restore on demand quickly and confidently. Testing can be as simple as restoring a single non-critical file on a network share drive or restoring an entire system. Testing should be done at least quarterly, if not more frequently, based on business appetite.
Organizational needs. An organization must match its business needs with backup schedules. Some may require more frequent backups as the data changes frequently or is business critical. Others may be able to survive to lose several weeks or months of data and still function. This comes down to business needs and abilities.
Finally, to ensure confidentiality of those backups, data at rest should be encrypted using approved encryption algorithms with checks and validation to ensure illegitimate changes are not made. I have seen instances where all virtual machine snapshots were being backed up by HPE Nimble. HPE Nimble is a very expensive solution, however they have proprietary encryption and do not allow the data to be altered while at rest. This means during a ransomware event; the backups cannot be encrypted by the malware. I have seen several businesses survive based on that piece of hardware and software. (This is not an endorsement, just what I have seen.)
Conclusion
Backups are extremely important to business operations. Many disasters can occur at any given moment. The best way to recover from disaster is to be highly prepared with a solid backup plan. This solid plan should reflect the risk appetite of potential data loss, should be regularly practiced, and the backup copies should be saved in multiple physically and logically different places.