Backup HowTo

Never knew about this 'dd' command; makes things so very simple!

dd may be adequate and simple, but I still prefer Ghost, because its compression is good and fast.

If you want to compress the data on-the-fly, you can use the following:

dd if=/dev/sda | gzip > disk.img.gz
dd if=/dev/sda | bzip2 > disk.img.bz2

Good idea, thanks for the tip

I prefer partimage over DD, it is closer to norton ghost and in my opinion it is better than DD.John

Do we need to have special binaries for dd command, it somehow does not exist on my system.. was wondering..

Try to login as a root user.

hello sir,

I have one question that if i make a dd for my ubutu machine is it will take whole disk image and can i restore the as it is where preiveous one to the other machine (new)

I have a gate way of ubuntu which is using in my office and i want to make a standby machine what which is to be same partition and same configaration
pls help me out

Thanking you

Shreedhar
shreedhar.hodlurkar@rediffmail.com
+919632022500

Hello, yes, dd command creates a whole partition backup to an image file.

Yes you are right, life is much easier using the dd command. not all use this command may be because they dont know about it

I had the same problem, I logged in now as root and it works fine. Thanks for your post..

informative .. am yet to try out this ..

I've cloned a 500G HDD from our server to a removable media (USB external 2.5" HDD) with this command:
# dd if=/dev/sda of=disk.img

I wanted to make sure my image was correct so I did this:
# cmp disk.img /dev/sda

and I've found that the image I'm getting is _not_ identical to /dev/sda. Errors crop up and a lot more frequently than I expected. In fact if you use the -l flag with the cmp command you can see just how many sectors come back with errors, and with my setup, it's rather shockingly high! I am not certain what is the cause for these large number of errors. I am suspecting the cable from the server to the external HDD unit. The external drive is sitting on top of the top server in a rack and the cable is extending down past about 5 servers to one of the servers below.

I am thinking that I am going to try to write a script that will clone, then compare then reclone those sectors that failed to clone properly the first time. But before doing that, is there any other suggestion? This is going to take a long time with a 500G drive.

There is one very serious problem with your text. You do not mention that the dd copying must be done on unmounted disks/partitions. If you clone your mounted partition and, during the copying process the system writes to it (which will almost certainly be the case due to various system activities), then you end up in a mess in the relevant blocks of your copy.

If you want to use dd, boot from a CD using a Linux distribution like Knoppix or boot from your distro installation disk and choose not install but rescue mode (some distros like Fedora provide this functionality) and then use dd to copy UNMOUNTED partitions.

To copy the contents of the mounted ones, use rsync instead.

See my post above. Data on disk are constantly changing due to system activity so comparing your image and mounted partition on a running computer makes no sense. Also, it is granted that you will have many problems restoring your partition from a copy made this way.

I believe the USB cable was most likely the cause of the errors. I managed to clone the HDD over the network to another server and the image seems to be error free. If you are on server 1 and you want to make a backup clone of /dev/sda on the HDD of server 2 (IP address 192.168.4.226), the method for cloning I used was as follows (note I am running a live CD from server 1, so /dev/sda is not mounted and can be reliably cloned):
# cat /dev/sda | gzip -c | ssh 192.168.4.226 "cat - > server.img.gz"
cat /dev/sda takes the entire /dev/sda device (the HDD you want to clone) and dumps it to STDOUT.
gzip -c takes STDIN, compresses it and pipes it to STDOUT
ssh 192.168.4.226 opens a connection to the other server (you may be asked to enter the password for that server), and pipes STDIN to the command line there. The command line is: cat - > server.img.gz
cat - dumps the input to STDOUT, which came from server 1 thru ssh and dumps it into the file server.img.gz.
I haven't tried it yet, but to restore the image, something like this should be run:
# ssh 192.168.4.226 "cat server.img.gz" | zcat >/dev/sda
Note that compression and decompression is occuring on the local server that houses /dev/sda that is being cloned. This means that compressed data is flowing over the network instead of uncompressed data reducing your network traffic.
Thirdly, if you want to check that the image is error free the following line should do it:
# ssh 192.168.4.226 "cat server.img.gz" | zcat | cmp -l /dev/sda
The cmp command will return differences between its STDIN (which comes from the uncompressed data stream from the image on the other server) and the "file" /dev/sda. If there are no differences, it should return nothing. If you get 2 columns of data returned from the cmp -l, then those are the bytes that differ between the 2 data streams.
If you want something a bit more mentally reassuring than the cmp -l function (that returns nothing if it succeeds) you can use md5 to compare a checksum from the 2 respective streams:
# cat /dev/sda | md5
will give you a check-sum for the local /dev/sda
# ssh 192.168.4.226 "cat server.img.gz" | zcat | md5
will give you a checksum for the image on the remote server.
If the 2 checksums match, then the 2 streams should be identical
Another point of interest is that I cloned a 500GB HDD, but the compressed image file is only <30GB. This indicates that most of the 500GB HDD is all zeroes and therefore compresses very well. The fact that the cmp worked for the network backup, but failed for the external HDD backup indicates that the most likely cause of the errors was the not-too-well shielded cable running from the server to the external HDD.

Can I using dd to clone a OS drive while it is running? I have a system that I would like to avoid shutdown to clone the drive. While the server is running, I would think that I can use dd to clone the OS drive to a new USB drive without shutdown the server. Am I right?
Thanks.

Hello! and thanks for this valuable information. Currently I have a red hat server with mirrored 0 set up. I have four HD.
1- [Hard Drive] --> Mirrored 0 set up --> [Hard Drive] (MAIN HARD DRIVES)

2- [Hard Drive] --> Idling --> [Hard Drive] IDLING DRIVES

What I want to do is to create an image from the MAIN HARD DRIVES which is the RH linux a long with applications, files and mounted in the other two hard drives [IDLING DRIVES]. in this way I will be able to store the IDLING DRIVES in case of a failure. My question is => How can I distinguish between the two hard drives that currently hold the operating system and applications AND the two IDLING DRIVES, and how can i copy the image directly to those drives(IDLING DRIVES). Wow!! this seems to be a hard task for my limited knowledge. I appreciate any help. Thank you, thank you, thank you.

I try this on my +100GB drive and it says the image file is too big to create. A little researched turned this solution, which uses the split utility to span the image across several different files:

cd /mnt/hdb/backp/
dd if=/dev/hda | split -b=2000m

This solves one problem but creates two more.

1) I can't specify a filename and location in the dd command. Anyone know how I can specify a filename and path when using split? Or, for that matter, split the file at 650mb so they can fit on CDs if I need them to? split -b=650m doesn't seem to work as the man page would suggest.

2) More importantly, how do I restore an image spanning many files?

Image files with dd are huge
because they copy empty space too.

The trick could be to write zeros on the empty space of your partition.
So, your compressed file will be very little.
But, how?

http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-comm...

Awesome, thanks for this guide =p

Hi everyone,

I am not as blessed to know how to use Linux well yet. Please pardon me for being such a newb, I'm trying to learn... I am a very good Windows user though I reckon. Currently, I have partitioned my hard disk into three partitions, each one of which has a windows operating system, namely Windows Xp, Vista, and Win 7. In other words I have set up a trible boot, and a nice bootloader which gives me a lovely colourfl menu to choose from.

The partition sizes are 3.5GB, 20GB and 30GB respectively, and 300GB unallocated space. I am planning to use this Unallocated space as storage. But for now, I want to back up the hard disk EXACTLY as it is, including the MBR - everything -

I don't want to back up individual partitions. I want to keep a (raw?) copy of the hard disk data, in an image, without saving the empty space, and use it to slap the system back to the way it was when s***t hits the fan, without having to reinstall windows three times and go through all the tweaking and installing drivers and programs again.

I tried Norton Ghost, but it will only allow me to clone individual drives/partitions, and the MBR doesn't work after I'm done.

I have Knoppix, and this has been my first true experience with Linux. How do I use it to do what you guys are talking about? Any comments or help is appreciated.

Hi. As a Windows user (most of the time) I couldn't answer how to solve this using Linux. However I would like to recommend Acronis True Image which is able to do this job very easy. Try it.

I am trying to copy a fairly new installation of linux because I've done some customizing already but the fact that my primary partition has like 105 gigs of space means that the image is always that big even though I have like 3 gigs of real data.

There needs to be a way to get rid of empty space in the image first, or like ottix said if you could write zeros and compress it on the fly maybe that would work. Anyways this seems to be a poor backup solution for me right now.

Thanks for the info though. I was looking to see how a person can back up entire disks and your guide is really good. I have also done backups like this in a mac through Disk Utility. It's GUI is really good and informative, and it also doesn't record empty space. Is there something similar for linux?

-Evan

Unless you have an amazing computer, you might want to use gzip over bzip2 for this. On my system, bzip2 is the bottleneck in this operation, compressing input at <2MB/s, whereas gzip achieves 10MB/s. That's the difference between "a pretty long time" and "hours and hours" on a huge disk image.

nice cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f zero.fill
or
http://www.partimage.org/

"You must be sure that the capacity of /dev/sdb5 is larger than /dev/sda1."

Just wondering, why must it be larger that the first? Why can't it be the same size? And how much larger should it be?
Sorry if these are silly questions.

I think it should be same or larger.

On newer distributions disk by ID is used.

This means if you are going to take that cloned hard drive you will have to tell grub to look for /dev/sda1 not /dev/disk/id/wd2500..........part 1

And the same thing for anything else grub references.

Also in /etc/fstab you have to fix that as well from a disk id to a disk partion.

just use the "dd if =/dev/sda of=/dev/sdb" by not specifying the partition number (ie: "sda1") you are instructing the command to copy the entire disk. Beware, though, that this will take for-frickin-ever for your large hard drive. Personally, I use it on a per-partition basis, as well as copying my MBR, so that if I screw up operating system B I don't have to reset operating systems A and C, but if I need to re-do the whole system I have my MBR on hand. you dig?

Linux is very light... it never hangs when data stored a lot..... I always ask my friends to use linux..

For greater compression speed, try pigz. Same format as gzip, but uses multiple cpu cores. I compared packing a multi-gigabyte file with gzip and pigz on my quad core, and pigz was ~3.5 times faster.

What does the zero filling command do here ?

I understand that it is supposed to fill the drive with zeros but I cannot understand how ?

It writes zeroes to a "zero.fill" file until it fills up all the free space on the disk, then deletes that file.
Pretty interesting idea.

I've construed a nice way to back up the stuff from my server directly into an ssh connection to my home machine. Works well, costs nothing and it does not clutter the disk space on my server. Check it out: How to have a rock solid Linux backup without pro budget.

Great tip, Vlatko! Thanks for sharing it with us.

Hmm... now we have zeros that don`t exist anymore

I believe you may have some luck with the count parameter of dd. Simply calculate bs*count so that it doesn't copy past the last windows partition.

This is assuming that your 300GB partition is the last partition on the disk.

That way when you restore the image, nothing in the last partition will be touched. Do make sure that you calculate the offset exactly though. Perhaps using fdisk to print the partition table.

What happened is that you began creating a file whose contents were nothing but zeroes, and kept on filling that file with zeroes until the disk was full. When you then delete that file, all you've done is delete the directory entry for that file; the zeroes themselves are still there, and thus pretty much all of the empty space on your disk has been zeroed out - which makes your compressed disk image all that much more compact.

To fill up unused space in partition with zeros use command:
zerofree -v /dev/sda2
note that the partition can not be mounted at that time.

To fill up partition with zeros you can use command:
dd if=/dev/zero of=/dev/sda4
this will erase any data in the partition /dev/sda4

If you are backing up whole disk which contains swap partition you might want to fill up that swap partition with zeros as well:
- with command 'fdisk -l' find your 'Linux swap' partition's device, e.g. /dev/sda1
- then zero the partition:
dd if=/dev/zero of=/dev/sda1

To backup whole disk:
bzip2 -c /dev/sda > /mnt/your.image.file.dd.bz2
the file will be compressed on fly.

MBR backup/restore from http://www.debianhelp.co.uk/ddcommand.htm
MBR backup:
In order to backup only the first few bytes containing the MBR and the partition table you can use dd as well.
dd if=/dev/hdx of=/path/to/image count=1 bs=512

MBR restore:
dd if=/path/to/image of=/dev/hdx
Add "count=1 bs=446" to exclude the partition table from being written to disk. You can manually restore the table.

An excellent set of instructions for cloning, but dd is very slow (circa 20 MB/sec) with the default block size of 512 bytes.

Most linux distro's will accept bs=1M, which is optimal for most modern PCs

I too was surprized to see no mention of dismounting the drives. I think it would be nice if the author would edit the article and include some if the best ideas in these replies, like umount, block size, and compression on the fly... But most certainly umount!!

And don't forget the use of a bootable os on DVD. Knoppix is free and one the best in the event you do not have a rescue disk. If u are wanting to backup the system disk , then umounting it while in the native operating system isn't going to go well!

1. Instant hard disk backup for data transfer or hard disk upgrading - hard disk clone
third party software for such feature:
http://www.easeus.com/disk-copy/
http://www.todo-backup.com/backup-resource/copy-and-clone-hard-drive.htm
http://www.partition-tool.com/copy-wizard/
2. Good habit of data security - preserve backup
Windows built-in backup feature: http://www.microsoft.com/windows/windows-7/features/backup-and-restore.aspx
third party software:
http://www.acronis.com/homecomputing/products/truimage/index.html
http://www.todo-backup.com/backup-resource/free-hdd-imaging-software.htm