Chapter 7. Backing up and restoring data using Acronis True Image Home There are many different products on the market today for backing up and restoring data, and we have already met some of them in

Data backup is an essential function for every responsible computer user. There is nothing worse than being aware, sitting in front of a computer, that the result of your monthly work or important files that cannot be restored have just been lost.

Data very often has a high price tag. And it doesn't matter if it's personal files or information that is used for work. Let's figure out what functions for data backup and recovery Microsoft developers have equipped the latest version of their operating system - Windows 10, and also consider the possibility of backing up data recovery using third-party programs.

Data backup: What is it?

File backup

We sometimes hear about the need to "create a computer backup". But to say this is not entirely correct, since there are two ways of backing up data. First, creating a backup copy of files, with which you can create copies of individual files and folders stored on a computer or any other storage medium. Every user who has important data on their computer should back it up on an external hard drive in case the original source is lost or deleted.

System backup

Another way to back up your data is known as System Backup or System Image. This is a more complex way of backing up data, since in this case we are talking about creating a copy of the entire operating system of a particular computer, along with programs, files and installations. A system backup is used to restore it in case Windows starts to work unstable or stops working altogether.

A Windows backup can take up hundreds of gigabytes of memory. But using special software for backing up data, you can configure the function to save only the changes that have been made since the last time the system was backed up.

Backing up and restoring data using Windows

Windows 10 has built-in tools that are designed to create backups of files and the system, as well as restore information with their help. Many users do not use them, not knowing about their existence or how to properly configure them. They tend to use third-party programs more often than tools built into the operating system.

File history

The most in a simple way creating a backup copy of files and folders, as well as restoring from them, is File History. With this function, you can copy data regularly, according to the schedule set by the user, and save it to external hard disk or other external storage medium. Copies of files can also be saved to your computer's hard drive (although this is contrary to the basic rule of backup).

To customize File History go to the Start menu and select Settings. In the Settings menu that opens, enter the Update and security / Backup service menu.

On the right side of the panel, under File History Backups, click Add Disk. You will be prompted to select one of the external storage media connected to your computer. Select the one to which you want to back up the data and set the backup parameters: the interval for saving copies of files and the duration of their storage, if necessary, you can specify a specific folder for a copy of files from which you want to create or remove unnecessary ones.

During the creation of a backup, File History also automatically copies all folders of the user account: Pictures, Documents, Downloads, etc.

This function can be turned off at any time, but in order for the copying of user-specified data to occur at a set time interval, Automatic file backup must be turned on.

By default, files are backed up every hour. The minimum time that can be set is 10 minutes, the maximum is Daily. You can also set the storage duration for backups: from one month until free space is needed (in this case, older backups will be deleted automatically to free up space for new backups).

Restoring files from a backup

To recover files from a disk with a data backup, go to the Control Panel and select the File History menu.

In the left column of the window that opens, select Recover personal files, go to the folder and find the file you want to recover. To restore a file, click the green button below and specify the path to restore it. You can select the backup versions of files by date and time by pressing the left and right arrow buttons.

OneDrive cloud service

Another way to protect yourself from losing important files is to transfer them to the cloud storage and synchronize it with your computer. There are many such storages: Dropbox, Google drive etc. But a feature of the latest versions of Windows (10, 8, 8.1) is the presence of Microsoft's own cloud service built into the operating system, which is called OneDrive.

When using one of the cloud services, they usually create a folder on the computer disk under their own name, the files from which they synchronize. V latest versions Windows folder OneDrive exists by default, i.e. the service is available to every user immediately after installing the operating system.

Any file. which will be saved in this folder will be immediately copied to the cloud storage and synchronized with it. The user can also access the files in this folder from any other device, just go to your OneDrive account from it.

Recovery Point

Another side of backup and recovery Windows data 10, there is recovery system files and the health of the operating system. Creating and restoring an operating system backup is a more complicated process than with personal files and user folders. But for this, Windows 10 provides a special function - System Restore. With its help, the user can restore the operating system to work, "rollback" to the saved early state - the Recovery Point.

By default, the restore function in the operating system is not activated. In order to be able to use it, it must be configured. System Restore works by creating Restore Points, with the help of which the state of Windows is saved at a specific point in time. Together with the settings and state of the operating system, the Recovery Point includes installed applications(For example, Microsoft Office) and device drivers (for example, a video card).

The user can create a restore point at any convenient time. It is also created automatically if an application is installed on the computer or system updates are downloaded, etc. But remember that System Restore does not restore the user's personal files.

If you begin to notice that the operating system is malfunctioning or with malfunctions and errors, then you can run one of the previously saved restore points and Windows will return to the state at the time of its creation.

To do this, go to the Control Panel and select the Restore / System Restore Settings / Configure menu, with which you can activate and configure the function.

To restore Windows from a restore point, go to the Control Panel and select the Restore / Run System Restore menu, select the desired Restore Point and click Next.

System image

Probably the most basic and complex Windows backup feature is a feature called System Image.

Using this option, you can create a "duplicate" of the entire computer disk, including Windows 10, as well as all programs and personal data of the user. The system image is saved to an external hard drive, from which, if necessary or if the computer fails, it is deployed back to the computer hard drive. As a result, the user receives a workable version of the operating system, along with programs and files.

This is very useful function, but its disadvantage is that the user does not have the ability to restore individual files from the system image, as from the File History. You can only fully expand the entire image. Therefore, users often use these functions in parallel.

To create a system image, go to Control Panel and select the File Backup and Restore menu. In the left column, click Create System Image, and select the drive you want to save it to. Please note that the System Image is usually large.

To restore your system from a System Image, go to System Settings / Update & Security / Recovery, and select Custom Boot Options. After Windows 10 restarts, select System Restore from an image from the menu provided.

Other backup and restore software Windows files 10

ALEXEY BEREZHNOY, System Administrator. Main areas of activity: virtualization and heterogeneous networks. Another hobby besides writing articles is the popularization of free software.

Backup
Theory and practice. Summary

To organize the backup system most effectively, you need to build a real strategy for saving and restoring information.

Backup (or, as it is also called, backup - from the English word "backup") is an important process in the life of any IT structure. This is a rescue parachute in the event of an unforeseen disaster. At the same time, backup is used to create a kind of historical archive of the company's business activities over a certain period of its life. Working without a backup is like living in the open - the weather can deteriorate at any moment, and there is nowhere to hide. But how to organize it correctly so as not to lose important data and spend fantastic sums on it?

Usually, articles on the topic of organizing backups deal mainly with technical solutions, and only occasionally attention is paid to the theory and methodology of organizing data storage.

This article will focus on just the opposite: the focus is on general concepts, and technical tools will be touched upon as examples only. This will allow us to abstract from hardware and software and answer two main questions: "Why are we doing this?", "Can we do it faster, cheaper and more reliable?"

Goals and objectives of backup

In the process of organizing a backup, two main tasks are set: recovery of infrastructure in case of failures (Disaster Recovery) and maintaining a data archive in order to subsequently provide access to information for past periods.

A classic example of a Disaster Recovery backup is an image of a server's system partition created by Acronis True Image.

An example of an archive can be a monthly unloading of databases from 1C, recorded on cassettes with subsequent storage in a specially designated place.

There are several factors that differentiate a backup for quick recovery from the archive:

• Data storage period. For archival copies, it is quite long. In some cases, it is regulated not only by business requirements, but also by law. Disaster recovery copies have a relatively small amount. Usually, one or two (with increased reliability requirements) backups for Disaster Recovery are created with a maximum interval of a day or two, after which they are overwritten with fresh ones. In particularly critical cases, it is also possible to update the disaster recovery backup more frequently, for example, once every few hours.
• Fast access to data. The speed of access to a long-term archive is not critical in most cases. Usually the need to "raise data for the period" arises at the time of reconciliation of documents, return to previous version etc., that is, not in emergency mode. Another thing is disaster recovery, when the necessary data and the performance of services must be returned as soon as possible. In this case, the speed of access to the backup is extremely important.
• The composition of the information being copied. The archived copy usually contains only user and business data for the specified period. The disaster recovery copy contains, in addition to this data, either system images or copies of operating system and application software settings, and other information required for recovery.

Sometimes it is possible to combine these tasks. For example, an annual set of monthly full "snapshots" of the file server, plus changes made during the week. True Image is suitable as a tool for creating such a backup.

The most important thing is to clearly understand why the reservation is being made. Let me give you an example: a critical SQL server crashed due to a disk array failure. In stock there is a suitable Hardware, so the only solution to the problem was software and data recovery. The company's management asks an understandable question: "When will it start working?" - and is unpleasantly surprised to learn that it will take four hours to recover. The fact is that throughout the entire service life of the server, only databases were regularly backed up without taking into account the need to restore the server itself with all the settings, including software the DBMS itself. Simply put, our heroes saved only the databases, and forgot about the system.

Let me give you another example. Throughout his career, the young specialist created a single copy of a file server running Windows Server 2003 using the ntbackup program, including data and System State into a shared folder on another computer. Due to the lack of disk space, this copy was constantly overwritten. After a while, he was asked to restore the previous version of the multi-page report, which was damaged when saving. It is clear that, having no archive history with Shadow Copy turned off, he could not fulfill this request.

Which conclusion follows: you need to use both types of backups: for disaster recovery and for archive storage. In this case, it is imperative to determine the list of copied resources, the execution time of tasks, as well as where, how and how long the backups will be stored.

With small amounts of data and not very complex IT infrastructure, you can try to combine both of these tasks in one, for example, make a daily full copy of all disk partitions and databases. But it is still better to distinguish between the two ends and choose the right means for each of them. Accordingly, a separate tool is used for each task, although there are universal solutions, like the same Acronis True Image package or the ntbackup program

It is clear that when defining the goals and objectives of backup, as well as solutions for implementation, it is necessary to proceed from the requirements of the business.

There are different strategies that can be used to implement a disaster recovery task.

In some cases, a direct system recovery to bare metal is required. This can be done, for example, using Acronis True Image bundled with Universal Restore. In this case, the server configuration can be returned to service in a very short time. For example, it is quite possible to lift a partition with an operating system of 20 GB from a backup copy in eight minutes (provided that the archive copy is available over a 1 Gb / s network).

In another case, it is more expedient to simply "return" the settings to the newly installed system, such as copying configuration files from the / etc folder in UNIX-like systems (in Windows, this is roughly equivalent to copying and restoring the System State). Of course, with this approach, the server will be put into operation no earlier than the operating system is installed and the necessary settings are restored, which will take much more long term... But in any case, the decision of what to be Disaster Recovery stems from the needs of the business and resource constraints.

The fundamental difference between backup and redundant backup systems

This is another interesting question that I would like to raise. Redundant equipment redundancy systems mean the introduction of some redundancy into the hardware in order to remain operational in the event of a sudden failure of one of the components. A great example in this case is a RAID (Redundant Array of Independent Disks). In the event of a single disk failure, information loss can be avoided and a safe replacement can be made, preserving the data due to the specific organization of the disk array itself (read more about RAID in).

I've heard the phrase: "We have very reliable equipment, there are RAID-arrays everywhere, so we don't need backups." Yes, of course, the same RAID array will save data from destruction if one hard drive fails. But here's from data corruption computer virus or it won't save you from inept user actions. It will not save RAID even if the file system crashes as a result of an unauthorized reboot.

Let me give you one example. There are times when events develop according to the following scenario: when a disk fails, the data is restored due to the redundancy mechanism, in particular, with the help of saved checksums... At the same time, there is a significant decrease in performance, the server freezes, control is practically lost. System Administrator not seeing any other way out, it reboots the server with a cold restart (in other words, it clicks on "RESET"). As a result of this live overload, file system errors occur. The best thing to expect in this case is the long run of the disk checker in order to restore the integrity of the file system. In the worst case, you will have to say goodbye to file system and be puzzled by the question of where, how and in what time frame it is possible to restore data and server performance.

You cannot avoid backups even if you have a clustered architecture. A failover cluster, in fact, maintains the operability of the services entrusted to it in the event of a failure of one of the servers. In case of the above problems such as, virus attack or data corruption due to the notorious "human factor", no cluster will save.

The only thing that can act as an inferior backup replacement for Disaster Recovery is a mirrored backup server with continuous data replication from the main server to the backup (according to the Primary  Standby principle). In this case, if the main server fails, its tasks will be taken over by the backup one, and you don't even have to transfer data. But such a system is quite costly and time consuming to organize. Do not forget about the need for constant replication.

It becomes clear that such a solution is cost-effective only in the case of critical services with high requirements for fault tolerance and minimum recovery time. As a rule, such schemes are used in very large organizations with a high commodity and money turnover. And this scheme is an inferior substitute for backup copying because it doesn't matter if the data is damaged by a computer virus, inept user actions or incorrect work applications, data and software on both servers can be affected.

And, of course, no redundant backup system will solve the problem of maintaining a data archive for a certain period.

Backup window

Performing a backup places a heavy load on the redundant server. This is especially true for the disk subsystem and network connections. In some cases, when the copying process has enough high priority, this can lead to unavailability of certain services. In addition, copying data at the time of making changes is associated with significant difficulties. Of course, there are technical means to avoid problems while maintaining data integrity in this case, but if possible, it is better to avoid such copying on the fly.

The way out when solving these problems described above suggests itself: to postpone the start of the process of creating copies for an inactive period of time, when the mutual influence of backup and other working systems will be minimal. This time period is called the "backup window". For example, for an organization operating on the 8x5 formula (five eight-hour work days a week), such a "window" is usually weekends and night hours.

For systems operating according to the 24x7 formula (all week around the clock), the time of minimum activity is used as such a period, when there is no high load on the servers.

Types of backup

To avoid unnecessary material costs when organizing a backup, as well as, if possible, not go beyond the backup window, several backup technologies have been developed, which are used depending on the specific situation.

Full backup (or Full backup)

It is the main and fundamental method of creating backups, in which the selected data array is copied in its entirety. It is the most complete and reliable backup method, although it is also the most expensive one. If it is necessary to save several copies of data, the total stored volume will increase in proportion to their number. To prevent such waste, compression algorithms are used, as well as a combination of this method with other types of backup: incremental or differential. And, of course, a full backup is indispensable when you need to prepare a backup for a quick system recovery from scratch.

Incremental copy

Unlike a full backup, in this case, not all data (files, sectors, etc.) are copied, but only those that have changed since the last backup. To find out the copy time, you can use different methods for example, systems running Windows operating systems use a corresponding file attribute (archive bit) that is set when a file has been modified and cleared by the backup program. On other systems, the date the file was modified may be used. It is clear that a scheme using this type of backup will be incomplete if you do not carry out a full backup from time to time. When restoring the system completely, you need to restore from the last copy created by Full backup, and then roll the data from the incremental copies one by one in the order of their creation.

What is this type of copying used for? In the case of creating archival copies, it is necessary to reduce the consumed volumes on storage devices (for example, to reduce the number of used tape media). It will also allow minimizing the execution time of backup tasks, which can be extremely important in conditions when you have to work in a busy 24x7 schedule or pump large amounts of information.

Incremental copying has one thing to be aware of. Step-by-step recovery brings back the needed deleted files during the recovery period. Let me give you an example. Let's say a full copy is performed on weekends, and an incremental copy on weekdays. A user created a file on Monday, changed it on Tuesday, renamed it on Wednesday, and deleted it on Thursday. So, with sequential step-by-step data recovery over a weekly period, we will receive two files: with the old name on Tuesday before renaming, and with a new name created on Wednesday. This happened because different incremental copies were stored different versions the same file, and eventually all variants will be restored. Therefore, when restoring data from an archive "as is" in a sequential manner, it makes sense to reserve more disk space so that the deleted files can also fit.

Differential backup

It differs from the incremental one in that the data is copied from the last moment of the Full backup. In this case, the data are placed in the archive "on a cumulative total". In systems of the Windows family, this effect is achieved by the fact that the archive bit is not cleared during differential copying, so the changed data is included in the archive copy until the full copy clears the archive bits.

Due to the fact that each new copy created in this way contains data from the previous one, it is more convenient for complete data recovery at the time of the disaster. To do this, only two copies are needed: the full one and the last of the differential ones, so you can bring the data back to life much faster than rolling all the increments step by step. In addition, this type of copying is free from the above-mentioned features of incremental, when, after a full restoration, old files, like the Phoenix bird, are reborn from the ashes. Less confusion arises.

But differential copying is significantly inferior to incremental copying in saving the required space. Since each new copy contains data from the previous ones, the total amount of backed up data can be comparable to a full copy. And, of course, when planning the schedule (and calculating whether the backup process will fit into a temporary "window"), you need to take into account the time it takes to create the last, "thickest" differential copy.

Backup topology

Let's consider what are the backup schemes.

Decentralized scheme

The core of this scheme is a network share (see Figure 1). For example, a shared folder or FTP server. A set of backup programs is also needed, from time to time uploading information from servers and workstations, as well as other network objects (for example, configuration files from routers) to this resource. These programs are installed on each server and work independently of each other. The undoubted advantage is the simplicity of the implementation of this scheme and its low cost. Standard tools built into the operating system or software such as a DBMS are suitable as copy programs. For example, it could be the ntbackup program for the Windows family, the tar program for UNIX-like operating systems, or a set of scripts containing built-in SQL Server commands to dump databases into backup files. Another plus is the ability to use various programs and systems, so long as they all can access the target resource for storing backups.

The downside is the sluggishness of this scheme. Since the programs are installed independently of each other, you have to configure each one separately. It is rather difficult to take into account the peculiarities of the schedule and allocate time intervals in order to avoid competition for the target resource. Monitoring is also difficult, the copying process from each server has to be monitored separately from the others, which in turn can lead to high labor costs.

Therefore, this scheme is used in small networks, as well as in a situation where it is impossible to organize a centralized backup scheme with the available means. More detailed description this scheme and practical organization can be found in.

Centralized backup

In contrast to the previous scheme, in this case, a clear hierarchical model is used that works on the client-server principle. In the classic version, special agent programs are installed on each computer, and the server module of the software package is installed on the central server. These systems also have a dedicated server management console. The control scheme is as follows: from the console we create tasks for copying, restoring, collecting information about the system, diagnostics, and so on, and the server gives the agents the necessary instructions to perform these operations.

This is how most popular backup systems work, such as Symantec Backup Exec, CA Bright Store ARCServe Backup, Bacula, and others (see Figure 2).

In addition to various agents for most operating systems, there are developments for backing up popular databases and corporate systems, for example, for MS SQL Server, MS Exchange, Oracle Database and so on.

For very small companies, in some cases, you can try a simplified version of a centralized backup scheme without using agent software (see Figure 3). Also, this scheme can be used if a special agent for the used backup software is not implemented. Instead, the server module will use existing services and services. For example, "scoop" data from hidden shared folders on Windows servers or copy files by SSH protocol from servers running UNIX systems. This scheme has very significant limitations associated with the problems of saving files open for writing. As a result of such actions, open files will either be skipped and will not be backed up, or copied with errors. There are various workarounds for this problem, for example, rerunning the job to copy only previously opened files, but none are reliable. Therefore, such a scheme is suitable for use only in certain situations. For example, in small 5x8 organizations with disciplined employees who save changes and close files before leaving home. To organize such a truncated centralized scheme, working exclusively in Windows environment, ntbackup works well. If you need to use a similar scheme in heterogeneous environments or exclusively among UNIX computers, I recommend looking towards Backup PC (see).

Figure 4. Mixed backup scheme

What is off-site?

In our turbulent and volatile world, events can occur that can cause unpleasant consequences for the IT infrastructure and business in general. For example, a fire in a building. Or a breakthrough of a central heating battery in a server room. Or banal theft of equipment and components. One of the methods to avoid information loss in such situations is to store backups in a location remote from the main location of the server equipment. In this case, it is necessary to provide quick way access to data required for recovery. The described method is called off-site (in other words, keeping copies off-site). Basically, two methods of organizing this process are used.

Writing data to removable media and moving it physically. In this case, you need to take care of the means of quickly delivering the media back in case of failure. For example, store them in a nearby building. The advantage of this method is the ability to organize this process without any difficulty. The downside is the difficulty of returning the media and the very need to transfer information to storage, as well as the risk of damaging the media during transportation.

Copying data to another location over a network channel. For example, using a VPN tunnel over the Internet. The advantage in this case is that there is no need to carry media with information somewhere, the disadvantage is the need to use a sufficiently wide channel (as a rule, this is very expensive) and to protect the transmitted data (for example, using the same VPN). The difficulties encountered in transferring large amounts of data can be significantly reduced by using compression algorithms or deduplication technology.

Separately, it should be said about security measures when organizing data storage. First of all, it is necessary to ensure that the data carriers are in a protected area, and about measures that prevent unauthorized persons from reading the data. For example, use an encryption system, conclude a non-disclosure agreement, and so on. If removable media is involved, the data on it must also be encrypted. In this case, the marking system used should not help an attacker in analyzing the data. It is necessary to use a faceless numbering scheme for marking the carriers of the names of the transferred files. When transferring data over the network, it is necessary (as already mentioned above) to use secure data transfer methods, for example, a VPN tunnel.

We have covered the main points when organizing a backup. In the next part, methodological recommendations will be considered and practical examples to create an efficient backup system.

1. Description of backup in Windows system, including System State - http://www.datamills.com/Tutorials/systemstate/tutorial.htm.
2. Description of Shadow Copy - http://ru.wikipedia.org/wiki/Shadow_Copy.
3. Acronis official website - http://www.acronis.ru/enterprise/products.
4. Description of ntbackup - http://en.wikipedia.org/wiki/NTBackup.
5. Berezhnoy A. Optimizing the work of MS SQL Server. // System administrator, No. 1, 2008 - P. 14-22 ().
6. A. Berezhnoy. We organize a backup system for small and medium-sized offices. // System administrator, No. 6, 2009 - P. 14-23 ().
7. Markelov A. Linux on guard of Windows. Review and installation of the backup system BackupPC. // System administrator, No. 9, 2004 - S. 2-6 ().
8. VPN description - http://ru.wikipedia.org/wiki/VPN.
9. Data Deduplication - http://en.wikipedia.org/wiki/Data_deduplication.

In contact with

Almost all of our customers who have implemented backup systems (DBS) think that this has solved all their problems. They did everything in their power to ensure that everything was backed up, and in the event of an accident, it was correctly restored. But it often happens like this: the company is faced with a serious problem, and the traditional backup system does not allow recovery in the time that the company considers to be the target. In fact, the SLA that the backup system must meet is not being met. Alas, during our work we have accumulated many sad examples confirming this. Below we will give two cases and give advice on what technical means will reduce the recovery time. Choosing cases, we stopped at examples related to databases, where the most critical information for business was stored.

Retail Challenges

Customer: large insurance company.

A brief description of the cause of the crash: personnel error, incorrect patch installation on Oracle.

Description of the problem

This is a large company that has a mature IT department and invests enough in its equipment and staff. Suffice it to say that the Oracle DBMS ran on two Oracle Exadata, distributed over two technological sites, with a well-developed DR solution and a configured backup system.

One sad day it was decided to install a patch on the Oracle DBMS. Unfortunately, the engineer did not read the instructions to the end: "What am I, I won't install the patch without a piece of paper ?!" - and did it wrong. The error was noticed a few hours later, when the DBMS began to behave strangely and report it in the logs. Then the engineer decided to roll back. This action finally immobilized both copies of the database (all changes were replicated on Standby) and corrupted all data.

The company was left without its main information asset - a database through which all business processes worked. The business practically stopped.

Solution

The customer decided to restore from a backup. At that time, restoring a 5 TB database (now ~ 15 TB) took - attention! - more than 30 hours! In total, after 1.5 days, the base was restored a day before the accident. But there was more data! Everything else was restored by programmers and staff from other systems of the company, from primary documentation (application forms, copies, scans). It took another 1.5 days of hard work.

Total

2 High-End Oracle Exadata systems, Oracle Standby, a working backup system and 3 !!! of the day complete downtime at incorrect installation patch. Was this permissible according to company regulations? Of course not.

The main problem: the lack of tools for quick recovery from logical errors.

How could you have avoided

To mitigate the consequences of such accidents, you need to move in two directions. On the one hand, make backups more often, and on the other, be able to quickly recover. The following products might help:

Oracle FlashBack- a technology that allows you to do not only "roll forward" of new data on backup system Oracle, but also rollback to the desired transaction. With such a scheme, it would be possible to roll back the system before the problems with the patch began, which would greatly facilitate data recovery.

Snapshot technology. Snapshots allow you to back up and restore data in seconds. At the same time, they have little effect on performance, and it is possible to take pictures quite often (for example, once an hour). Thus, it was possible to roll back an hour and recover only an hour of lost data.

Continuous Data Protection- continuous data protection. This is a proprietary device or software that allows you to log all records with the ability to rollback to any point in time. Acts like Oracle FlashBack, but for any data.

Case: Hardware failure

Customer: Federal service in one of the subjects of the Russian Federation

A brief description of the cause of the crash: a hardware error inside the disk array.

Description of the problem

This time the company has a slightly less developed IT infrastructure, but it is more common among our customers: mid-level disk arrays, Oracle DBMS, Standby are not used.

As often happens, on Friday, when everyone was already happily going home, there was a hardware failure of the array. Due to a bug in the firmware, if the disk fails, the array turned the data into a mess. Because of this, the databases of the federal level service stopped working. For more than a day, the customer was waiting for a solution from the storage vendor. After analyzing all the logs, the vendor gave his conclusion: the data is lost!

Solution

The customer made the decision to restore from a backup. This process took about a day, despite all the tweaks and performance tuning (the base is quite large). While the database was being restored, the backup copy of the logs was lost (the Retention Period was too small, the SRK deleted them itself).

Further - deeper. The company, like many others, at some point used non-logged operations in Oracle, which seriously improves performance, but leaves no chance of recovery except from a backup. That is, it must be done immediately after passing the session of operations. Naturally, this was forgotten in the maintenance department over the years. Thus, part of the data was completely lost.

It took several more days to fully recreate the infrastructure services - there were no backups of operating systems, binaries, configurations, etc.

All the lost information was collected from primary documents (third-party databases, paper documents, data on the computers of tellers), which took another 3 days. Some documents may have never been recovered.

Total

The array problem caused data loss and downtime for about a week! In modern conditions, this can lead to bankruptcy of the company.

Main problems:

• The IBS was configured incorrectly, and no trial restorations were performed.
• There were no means of prompt recovery in the event of a disaster and redundant systems.
• There was no clear DR plan.

How could this have been avoided:

• Use Oracle Standby located on a different array. This would allow for a short time to switch to a running data instance.
• Oracle ZDLRA would allow to restore the database on the backup equipment in a much shorter time.
• Smart planning of backup and recovery processes could avoid such large losses and recover in less than a day.

Conclusion. From the examples above, it can be seen that the backup systems were installed and configured, but despite this, they did not succeed in recovering within the timeframes specified in the SLA.

The main problems of backup systems

Based on our experience, we decided to highlight a number of problems that, in our opinion, should be paid special attention to.

Backup and restore speed

On the this moment backup speed is directly proportional to the amount of data, while all our customers have an annual data growth of at least 30%. In 3-4 years, the data at least doubles, but for some companies this figure is even higher, while the backup speed does not change at all during the same time. Here we can make a simple conclusion that those terms and those SLAs that were relevant 3-4 years ago now need to be increased at least twice. At the same time, business requirements for data recovery (RPO / RTO) are constantly growing.

Gradually, all business processes of the company are transferred to IT and the paper primary (copies and originals of documents, statements, scans, etc.) are dying away. Everything revolves inside IT systems, and data loss is, in fact, the loss of everything. IT no longer has room for error. In the cases that we have given, all the time, while the data was not available due to various circumstances, the companies could not function. This led to both direct losses, when it was impossible to carry out the main business process of the organization, and to implicit ones, for example, to reputation losses, which are not so easy to measure in monetary terms, but which in the long term can cause no less damage to the company.

On the image I have reflected my observations regarding recovery time (RTO). As your data grows, the actual recovery time is bound to grow, and the SLA requirements only get stricter. The point on the graph where the actual time is equal to the required time has already been passed for most customers.

Recovery time versus data volume

Low granularity of recovery

In fact, most errors are related to the loss of some part of the data. At the same time, traditional backup tools allow you to restore data directly from backup, but more often you have to restore the entire system. If your database is 15 TB, you will spend several days on this. We do not know customers who have an RTO (Recovery Time Objective) requirement of 2 days. In our practice, there were no such examples when a client would say: “Guys, it's okay to recover in 2 days, I will tolerate it” - if the administrator accidentally deleted several rows from the database. A fairly common problem faced by our clients: how to extract a small piece of data from a backup without restoring it (and not spending several days on it).

Excessive RPO (Recovery Point Objective)

In a world where the primary paper has disappeared and everything is stored in IT systems, every second creates data that I would like to immediately protect - at the very moment when it was created. But this cannot be done using classic backup systems. For each piece of data there is a certain long period of time during which this data exists all over the world in a single copy. Our customers want to protect their data continuously, from the moment it appears. When deciding to restore from a backup, you will most likely have to recover a day ago, then the data will need to be obtained from somewhere else in a day. As a rule, it is long work administrators, taking several days. With the most negative development of events, this can result in a loss. critical information... Of course, the issue is not limited only to backup, it concerns the construction of the IT system as a whole, but the topic of the SRK in this case is very important, it cannot be neglected.

Hidden errors

Unfortunately, there are still no cheap and quick options to check how well the backup was made. Of course, this can be done with periodic test restores, but this is a very expensive operation in terms of human effort and IT resources. This is the work of a separate team on a separate hardware.

Alas, most of our clients do not do this. It often happens that everyone makes backups, but by the time of restoration, it turns out that they could not have been done - they simply cannot be restored, despite the outwardly correct operation of the RMS. This happens for various reasons. This is best illustrated with an example. One of our customers used a SAP system with an Oracle database. The backup was carried out by the built-in SAP tools with the help of one of the largest SRK vendors.

Two different backup policies were configured: one of them was file-based - it copied the data of operating systems and software settings, and the second - the database itself. Since they were directed to the same system, an exclusion list was configured and entered into the database. The file policy took this list into account and did not reserve the directories in which the database was located. Due to the peculiarities of the SRK architecture, the database backup policy ignored the list of exclusions and correctly copied the required data.

In one of the software releases, this vendor fixed this "error", from that day on, both politicians began to take into account the list of exclusions and bypass the database. Moreover, this did not in any way affect the errors in the SRK software, since it worked normally: all data not indicated in the list was backed up normally. The system reported that it was working properly.

Thus, everything worked for more than six months. Until that moment, until it was necessary to recover ...

Non-systematic approach

An important problem is the non-systematic approach to the problem of backup. The SRK has historically been built either by the company itself or by an integrator. At the time of construction, it certainly met all the requirements and performed its function entirely. Over the years, the company's IT landscape has changed. At the same time, the backup system simply adjusted to it as the system developed, and most often no systematic approach that would take into account the importance of the system's compliance with the initial indicators at all subsequent stages was not followed. When building a data protection system in your organization, remember - this is only part of your data protection strategy.

We have presented several case studies that demonstrate that the approach to data protection should be comprehensive. Alas, the SRK is just a reserve parachute, not a silver bullet, so when starting to create it, you need to clearly understand what place it will take in the global strategy for data protection.

To check how systematically you approached the issue of building the IMS, answer a few simple questions:

• Do you have a built-up risk model, within the framework of which the place of the IBS is spelled out?
• What failures does IBS protect you from?
• How do you protect yourself from other risks (these can be not just technical solutions, but also other compensatory measures)?
• Are you confident that the system will recover on time?
• Have you tested this in practice?

Solution

Based on our own experience and the experience of our customers, we tried to develop an approach that would solve or significantly reduce the consequences of the listed problems. The essence of our approach:

First, it is necessary to decouple the speed of backup and recovery from the volume of the system. Manufacturers of data storage systems, application software, and RMS suggest using some tools that can be used to solve this problem. Below I will describe the most promising of them.

Snapshots (snapshots) allow you to back up and restore data in seconds with little or no impact on performance. This is done by means of the array, and at the same time the SRK can be controlled, be part of its policy. This kind of backup and recovery really takes seconds, which distinguishes this technology from classical systems with alienable media.

Another solution could be to use various application tools such as Oracle Standby, DB2 HADR, MS SQL Always On. All of these tools allow you to have a working copy of a production system, detached from the original, which can be deployed instantly. This allows you to start working immediately after failures.

The second is to make it possible to restore only the data you need. Our approach takes into account that when recovering a part of the data, we do not need to copy the entire system as a whole, we can recover the data that we need at the moment. This is achieved by the ability to quickly deploy or use already deployed systems that contain this data. As in the first case, snapshot allows you to solve this problem (you can quickly open a snapshot to a neighboring server and pull out the required piece of data). This also includes continuous data protection technologies, for example, Oracle Standby with Flashback, continuous data protection (CDP) solutions. They allow you to quickly deploy a working copy of your data to the right moment time.

When you need to get one logical block, for example, a row or a database table, these tools greatly facilitate the task, allowing you to recover the necessary piece of data without restoring the entire copy.

The third is to reduce the gap between the appearance of data and their protection. This can be achieved in several ways, based on the specifics of a particular case and the degree of importance of the data.

For example, for less critical systems, the time interval for backups can be reduced to a few hours. In this case, we use snapshots. They can serve as a restore point that can be done once an hour. Some modern arrays handle these processes well enough and can store enough a large number of system snapshots. This is a great way out of a situation when you need to roll back for a while.

For the most critical systems, there may not be a time slot at all - the data must be protected continuously. There are several solutions of this class, for example, Oracle Standby with FlashBack, which allows you to roll back the database for a while by logging all changes. You can also use the Oracle ZDLRA PAK, which almost synchronously receives all changes in the database, or hardware and software systems general purpose such as EMC RecoverPoint, Vision Solutions Double-Take software. They also log all changes and allow you to recover to any point in the time interval.

Oracle Zero Data Loss Recovery Appliance (ZDLRA) should be mentioned when it comes to innovation in backup and recovery systems. This appliance from the Oracle Engineered Systems family provides backup and rapid recovery capabilities for Oracle Database on any platform and any Edition (Enterprise and Standard). ZDLRA is based on virtual backup databases (Virtual Full Backup), obtained on the basis of the first full backup and subsequent change logs. Due to these virtual backups, it is possible to restore the database to any point in time much faster than with the classic use of the SRK according to the "once a week full backup, once a day incremental" scheme. The ZDLRA can be said to be continuing the direction set by Oracle Exadata. Exadata uses special software to implement an innovative storage system optimized for Oracle Database tasks. And in ZDLRA there is a special Software that optimizes the backup of the Oracle Database.

Now we are only talking about quick recovery. In the event of major disasters or the need to recover a longer time ago, conventional backups remain an indispensable tool. But in the current conditions it is only a reserve parachute, deployed at the last moment.

The fourth is to reduce hidden errors. There is only one way to make sure that the backup is working correctly - to try to restore it. This is the most correct and rarely used method by our customers.

But we offer a way out of this situation as well. First, have easily recoverable instances of systems. This is again a story about snapshot and standby systems, which can be quickly deployed and tested. It will take incomparably less time and effort than “unwinding” the entire backup. Of course, this does not always help, but it leaves a little more hope that in the event of an emergency it will be possible to restore data at least by these means.

Secondly, some SRCs allow automated testing. At a certain time on the schedule, you can run virtual machines in an isolated environment and using predetermined algorithms to check whether the data has actually been restored, whether the application is available, whether it is consistent, and whether it responds to the necessary requests. In this way, administrators can get rid of long routine work.

Fifth, the transparency of the backup system. The described integrated approach involves the construction of a complex system using a variety of technologies from different manufacturers. The task of making this system really workable, to lay in it the possibility of further changes and scaling, is nontrivial, and it can be solved in two ways:

• The first way is provided that the customer is competent enough himself and wants to take this system into operation. Here, as an integrator, we help build all the necessary processes, create a regulatory framework, develop all the necessary instructions and plans so that the customer's IT department can further independently develop and operate the system in the right track... And then transfer all this practical base of regulations and tasks to the customer in the form of a working system of business processes.

• The second way, when the customer is not sure that he will be able to maintain the SRK system constantly in a combat state, the solution will be to transfer the system to partial or full outsourcing. And we have such clients who successfully use this service, constantly increasing both the SLA requirements and the scale of our involvement as an IT outsourcer.

Unfortunately, there is no universal recipe yet that would solve the problem of data recovery in the current conditions of constant growth and complication of systems. Only a combination of the above solutions and a systematic approach will allow companies to recover data in the time required by the business.