Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Unix Tape Archiver (Tar)

Tar is essentially a very old (dates back to Version 6 of AT&T UNIX, circa 1975) zero-compression archiver and generally can be replaced with any archiver that has a zero-compression option (for example zip).  It is often used along with gzip and bzip2.  Solaris tar understands ACLs. Not sure about GNU (on Solaris it does not). But as a zero compression archiver tar has several capabilities that regular archivers usually do not have and as such it is convinient for many tasks like backups and replication of filesystem from one server to another.

File limits are different for various OSes. Often Os itself has 2G file limit.  Obviously, older tar programs also won't be able to handle files or archives that are larger than 2^31-1 bytes (2.1 Gigabytes). Try running 'tar --version'.  If the first line indicates you are using gnu tar, then any version newer than 1.12.64 will in principle be able to provide you with large files. Try  command:

 strings `which tar` | grep 64

you should see some lines saying lseek64, creat64, fopen64. If yes, your tar contains support for large files. GNU tar docs also mention that the official POSIX tar spec limits files to 8GB, but that gnu tar will generate non-POSIX (therefore possible non-portable files) with sizes up to something like 2^88 bytes. So tars that you want to use on any other POSIX computer are limited to 8GB files.

Since the majority of  tar files are gzip'ed, the maximum filesize may be limited to 2 GB due to gzip limitation. This is the case if you use gzip version 1.2.4 and below. The newer versions (1.3.5 and above) support large files. To check gzip version use gzip --version.

See also Google Answers UNIX Question! tar size constraint.

The limit for file names is around 256 but can be as low as 100 on older Oses.

If you want larger archives you probably need to use star

Note:

In old versions of tar because of limitations on header block space in the tar command, user numbers (UIDs), and group identification numbers (GIDs) larger than 65,535 will be corrupted when restored by GNU tar. 

GNU tar has several interesting additional features like:

• -d, --diff, --compare find differences between archive and file system
• -p, --same-permissions, --preserve-permissions extract all protection information
• -P, --absolute-names don't strip leading '/'s from file names

Tar is one of the few backup programs that is pipable.  The tar command can specify a list of files or directories and can include name substitution characters. The basic form is

tar keystring options outputfile filenames....

The keystring is a string of characters starting with one function letter (c, r, t , u, or x) and zero or more function modifiers (letters or digits), depending on the function letter used.

The functions supported by tar are the same as for any archiver:

• c to create a new archive
• t Table of Contents. The names of the specified files are listed each time they occur in the tarfile. If no file argument is given, the names of all files in the tarfile are listed. With the v function modifier, additional information for the specified files is displayed.
• x  Extract or restore. The named files are extracted from the tarfile and written to the directory specified in the tarfile, relative to the current directory. Use the relative path names of files and directories to be extracted. If a named file matches a directory whose contents has been written to the tarfile, this directory is recursively extracted. The owner, modification time, and mode are restored (if possible); otherwise, to restore owner, you must be the super-user.
• u  Update. The named files are written at the end of the tarfile if they are not already in the tarfile, or if they have been modified since last written to that tarfile.  An update can be rather slow. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers.
• r Replace. The named files are written at the end of the tarfile. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers.

The most popular options include:

• v toggles verbose output
• f name Use the tarfile argument as the name of the tarfile. If f is specified, /etc/default/tar is not searched. If f is omitted, tar will use the device indicated by the TAPE environment variable, if set; otherwise, it will use the default values defined in /etc/default/tar. If the name of the tarfile is '-', tar writes to the standard output or reads from the standard input, whichever is appropriate. tar can be used as the head or tail of a pipeline. tar can also be used to move hierarchies with the command:

Creation of the tar archives

The tar command archives and extracts files to and from a single file (tarfile). A tarfile historically was a magnetic tape, but now it's usually a disk file. It can be any file.  tar's actions are controlled by the key argument, which should contain no spaces. Function modifier arguments are listed on the command line in the same order as their corresponding function modifiers appear in the key string.

The function portion of the key is specified by one of the following letters:

c  - Create. Writing begins at the beginning of the tarfile, instead of at the end.

Most often, the tar command for creating a tar archive uses a string of options as well as a list of what files are to be included and names the device to be written to. The default device, /dev/rmt0, is seldom used today, the most common is to archive into file that is often processed additionally by gzip

tar cvf myarchive.tar * 

The cvf argument string specifies that you are writing files (c for create), providing feedback to the user (v for verbose), and specifying the device rather than using the default (f for file). You can instantly pipe tar archive into gzip by specifying z option:

tar cvzf myarchive.tgz  *

The convention is to extension tar for tar archive and tgz for tared and gziped archives. tgz archives are monolithic bricks and are just designed to store files. But you can do a lot of operations with tar archives. In the past instead of gzip, the standard compress utility was used  In this case archives have prefix tar.Z.

The tar command also enables you to create lists of files that should be included or excluded from the archive. The options -I and X represent include and exclude from the list of files. For example

ls *.zip > exclude

tar cvf myarchive.tar -X exclude *
 

in this case zip files will not be included into the archive. It's a good idea to exclude the exclude file itself, as well as the tar file that you are creating in your exclude file. Notice that this has been done in the following example:

tar cvf myarchive.tar -X exclude *

Similarly, the include file can be used to specify which files should be included. In the next example, both an include and an exclude file are used. Any file that appears in both files, by the way, will be included.

tar cvf myarchive.tar -X exclude -I include

Notice haw we use options that requre pararmeters. the first such option is used as the last in in the first string of options (cvf) and then each option soecified with its parameter.

Combining tar with the find utility, you can archive files based on many criteria, including such things as how old the files are, how big, and how recently used. The following sequence of commands locates files that are newer than a particular file and creates an include file of files to be backed up.

find -newer lastproject -print >> include

tar cvf myfiles.tar -I include

That is especially important if you bring archive to a new place after modifing some files in the morning and need to merge changes back inthe evening. GNU tar has some features that enable it to mimic the behavior of find and tar in a single command.

Listing of the archives

To list the contents of a tar file without extracting, use the t option as shown below. Including the v option as well results in a long listing.

tar tf myfiles.tar

tar tvf myfiles.tar

Extraction of the tar archives

Archives created with tar include the file ownership, file permissions, and access and creation dates of the files. The p (preserve) option restores file permissions to the original state. This is usually good since you'll ordinarily want to preserve permissions as well as dates so that executables will execute and you can determine how old they are. In some situations, you might not like that original owners are retrieved, since the original owners may be people at some other organization altogether. The tar command will set up ownership according to the numeric UID of the original owner. If someone in your local passwd file or network information service has the same UID, that person will become the owner; otherwise the owner will display numerically. Obviously, ownership can be altered later.

 tar xvpf  myachive.tar

Extract each file from a shell prompt by typing tar xvzf file.tar.gz from the directory you saved the file.

Tar Usage in Pipes, "Back-to-back tar"

mv command could not be used to move directories across file systems. A file system can be thought of as a hard drive or hard drive partition. The mv command works fine when you want to move a directory between different locations on the same file system (hard drive), but it doesn't work well when you want to move a file across file systems. Depending on your version of mv, an error message could be generated when you try to do this.

For example, consider this directory:

$ ls -F /tmp/ch22
ch22-01.doc ch22.doc@

If you use the mv command to move this directory in the directory /home/ranga on a different file system, an error message similar to the following is generated:

mv: cannot move 'ch22' across filesystems: Not a regular file

Some UNIX versions implement a workaround inside mv that executes the following commands:

$ rm -rf destination
$ cp -r source destination
$ rm -rf source

Here source and destination are directories.

The main problem with this strategy is that links in the source directory are not always copied correctly. Most of the time, the file that the link points to is copied instead of the link itself. In the case of the directory /tmp/ch22, you would end up with two copies of the file ch22-01.doc, which is not desirable.

In addition to this, there are two other minor problems with using cp:

• Some versions of the cp command do not copy a file's owner and group. With these versions of cp, the copied file has a different owner and group than the original.
• Some versions of cp do not copy a file's permissions correctly. With such a version of cp, the copied file might have different permissions than the original.

The workaround for these problems is to use the tar ( tar as in tape archive ) command to copy directories. This is usually accomplished with pipe that became a Unix idiom:

(cd mydata; tar cvf - *) | tar xvpBf -

What this command does is move to a subdirectory and read files, which it then pipes to an extract at the current working directory. The parentheses group the cd and tar commands so that you can be working in two directories at the same time. The two - characters in this command represent standard output and standard input, informing the respective tar commands where to write and read data. The - designator thereby allows tar commands to be chained in this way. The following command is similar, but it reads the files from the local system and extracts them on a remote host:

tar cvf - mydata | rsh server01 "cd /pub/data; tar xvpBf -"

Archives created with tar include the file ownership, file permissions, and access and creation dates of the files. Once the files are extracted from a tar file they look the same in content and description as they did when archived. The p (preserve) option will restore file permissions to the original state. This is usually good since you'll ordinarily want to preserve permissions as well as dates so that executables will execute and you can determine how old they are. In some situations, you might not like that original owners are retrieved, since the original owners may be people at some other organization altogether. The tar command will set up ownership according to the numeric UID of the original owner. If someone in your local passwd file or network information service has the same UID, that person will become the owner; otherwise the owner will display numerically. Obviously, ownership can be altered later.

Tar archives can be transferred with remote copy commands rcp, ftp, kermit, and uucp. These utilities know how to deal with binary data. You will generally not use tar when mailing archived files, but you can first encode the files to make it work. The uuencode command turns the contents of files into printable characters using a fixed-width format that allows them to be mailed and subsequently decoded easily. The resultant file will be larger than the file before uuencoding; after all, uuencode must map a larger character set of printable and nonprintable characters to a smaller one of printable characters, so it uses extra bits to do this, and the file will be about a third larger than the original.

The second system can be remote is you use rsh. This is often called tar-to-tar file transfere. One tar command creates an archive with one tar command while the other extracts from the archive without ever creating a *.tar file. The only problem with this command is that it looks a bit awkward and, because the command depends on rsh, it requires that one of the systems trust the other (a potential security risk) when moving files between systems.

To copy a directory hierarchy using this technique, first position yourself in the source directory:

% cd fromdir

Next, tar the contents of the directory using the create (i.e., the "c" Option). Pipe the output to a tar extract (i.e., the "x" option) command. The tar extract should be enclosed in parentheses and contain two parts: 1) the cd part, which positions you in the destination directory, and 2) the extract part, which extracts the files and supplies excessive output in the process (due to the verbose, "v", option).

% tar cBf - * | (cd todir; tar xvpBf -)

The hyphens in the tar command inform tar that no file is involved in the operation. The B forces multiple reads and allows the command, as needed, to work across a network. The "p" is the preserve option - generally the default when the superuser uses this command.

Using tar-to-tar commands across a network is a little messy, but the syntax would look more or less like this:

cd boson-backups

rsh boson "cd /export/home/slee; tar cBf - *" | tar xvBf -

Notice how we group the remote commands to clearly separate what we are running on the remote host from what we are doing locally. You might also use tar in conjunction with dd to read files from, or write files to, a tape device on a remote system. In the following command, we copy the files from the current directory and write them to a tape device on a remote host.

tar cvfb - 20 * | rsh boson dd of=/dev/rmt0 obs=20

Back-to-back tar commands have been used for many years to copy or back up files.

Old News ;-)

Google Answers UNIX Question! tar size constraint.

Posted: 12 Jun 2002 23:47 PDT
Expires: 19 Jun 2002 23:47 PDT
Question ID: 25116

What is the size constraint to "tar" in a UNIX or Linux environment?

Subject: Re: UNIX Question! tar size constraint.
Answered By: philip_lynx-ga on 13 Jun 2002 01:05 PDT
Rated:

Hi pwharff,

The quick answer is: 2^63-1 for the archive, 68'719'476'735 (8^12-1)
bytes for each file, if your environment permits that.
 
as I understand your question, you want to know if you can produce tar
files that are biger than 2 GBytes (and how big you can really make
them). The answer to this question depends on a few simple parameters:

1) Does your operating system support large files?
2) What version of tar are you using?
3) What is the underlying file system?

You can answer question 1) for yourself by verifying that your kernel
supports 64bit file descriptors. For Linux this is the case for
several years now. A quick look in /usr/include/sys/features.h will
tell you, if there is any line containing 'FILE_OFFSET_BITS'. If there
is, your OS very very probably has support for large files.

For Solaris, just check whether 'man largefile' works, or try 'getconf
-a|grep LARGEFILE'. If it works, then you have support for large files
in the operating system. Again, support for large files has been there
for several years.

For other operating systems, try "man -k large file', and see what you
get -- I'll be gladly providing help if you need to ask for
clarification to this answer. Something like "cd /usr/include; grep
'FILE_OFFSET_BITS' * */*" should tell you quickly if there is standard
large file support.

2) What version of tar are you using? This is important. Obviously,
older tar programs won't be able to handle files or archives that are
larger than 2^31-1 bytes (2.1 Gigabytes). Try running 'tar --version'.
If the first line indicates you are using gnu tar, then any version
newer than 1.12.64 will in principle be able to provide you with large
files. Try to run this command: "strings `which tar`|grep 64", and you
should see some lines saying lseek64, creat64, fopen64. If yes, your
tar contains support for large files.

If your tar program does not contain support for large files (most
really do, but maybe you are working on a machine older than 1998?),
you can download the newest gnu tar from ftp://ftp.gnu.org/pub/gnu/tar
and compile it for yourself.

The size of files you put into a tar archive (not the archive itself)
is limited to 11 octal digits, the max. size of a single file is thus
ca. 68 GBytes.

3) Given that both your operating system (and C library), and tar
application support large files, the only really limiting factor is
the file system that you try to create the file in. The theoretical
limit for the tar archive size is 2^63-1 (9'223'372 Terabytes), but
you will reach more practical limits (disk or tape size) much quicker.
Also take into consideration what the file system is. DOS FAT 12
filesystems don't allow files as big as the Linux EXT2, or Sun UFS
file systems.

If you need more precise data (for a specific file system type, or for
the OS, etc.) please do not hesitate to ask for clarification.

I hope my answer is helpful to you,

--philip

ftp tar file size limit - Linux Forums

rockytopvols Just Joined!     Join Date: Sep 2006 Posts: 2

ftp tar file size limit? I am trying to back up my linux box to my windows box's hard drive. To do this I am using the Knoppix distro to boot my linux box. Then I am taring and ftping every file and sending it to my windows box through ftp. (I wanted to tar the files first, so I can preserve permissions) On my windows xp box I am running filezilla's ftp server, and I am transfering to an external external 320Gb NTFS formated hard drive attached to to it through usb. I don't have enough space left on my linux box to tar everything and then transfer, so I am using the following commands: ftp 192.168.1.101 21 binary put |"tar -cvlO *.*" stuff.tar It always stops transfering just before 2Gb (1,972,460KB), and the file should be 20Gb or so. What am I doing wrong? Is there some file size limit that I don't know of for ftp or tar? The NTFS files systems should allow bigger files from what I have read. I couldn't find any limit for filezilla. Is this the right place to ask? Thanks

rockytopvols

View Public Profile

Send a private message to rockytopvols

Find More Posts by rockytopvols

Sponsored Links

09-25-2006	#2 (permalink)
Marsolin Linux Newbie     Join Date: Aug 2006 Posts: 222	I believe NTFS has a 2GB file limitation unless you are running a storage driver with 44-bit LBA support. __________________ Chad http://linuxappfinder.com

 

09-25-2006	#3 (permalink)
rockytopvols Just Joined!     Join Date: Sep 2006 Posts: 2	Everywhere I have read the NTFS limit is in the tens of Terabytes range. I have some files that are bigger than that now.

 

02-05-2007	#4 (permalink)
nlsteffens Just Joined!     Join Date: Feb 2007 Posts: 1	tar file size limit Generally, tar can't handle files larger than 2GB. I suggest using an alternative to tar, 'star'. A more comprehensive answer is available here: http://answers.google.com/answers/threadview?id=25116 By the looks of it, gnu tar versions newer than 1.12.64 can handle large files but I can't confirm this. Regards, Nick

 

02-13-2007	#5 (permalink)
Alex Stan Just Joined!         Join Date: Aug 2006 Location: Hamilton, Ontario Posts: 88	I have a similar problem with big files: I have a 2.2 Gig file on a linux computer. And i mounted Shared Documents(smbfs) from a another(windows) computer. So when i try to copy it it stops at 2 GB. I even tried moving the file in apache, so i can download the file, but apache won't let me. I can't archive it either. Is there any way to move that file? __________________ I like linux!

 

05-30-2007	#6 (permalink)
sbhagat Just Joined!     Join Date: May 2007 Posts: 3	If you are using smbclient, then follow the kbase article: redhat.com | Knowledgebase Regards, Subodh Bhagat

ONLamp.com Backing up Files with Tar

One last thing about creating archives with tar: tar was designed to back up everything in the specified directory. This means that every single file and subdirectory that exists beneath the specified directory will be backed up. It is possible to specify which files you don't want backed up using the X switch.

Let's say I want to backup everything in the www directory except for the apache2 and zope subdirectories. In order to use the X switch, I have to create a file containing the names of the files I wish to exclude. I've found that if you try to create this file using a text editor, it doesn't always work. However, If you create the file using echo, it does. So I'll make a file called exclude:

echo apache2 > exclude
echo zope >> exclude

Here, I used the echo command to redirect (>) the word apache2 to a new file called exclude. I then asked it to append (>>) the word zope to that same file. If I had forgotten to use two >'s, I would have overwritten the word apache2 with the word zope.

Now that I have a file to use with the X switch, I can make that backup:

tar cvfX backup.tar exclude www

This is the first backup I've demonstrated where the order of the switches is important. I need to tell tar that the f switch belongs with the word backup.tar and the X switch belongs with the word exclude. So if I decide to place the f switch before the X switch, I need to have the word backup.tar before the word exclude. This command will also work as the right switch is still associated with the right word:

tar cvXf exclude backup.tar www

But this command would not work the way I want it to:

tar cvfX exclude backup.tar www
tar: can't open backup.tar : No such file or directory

Here you'll note that the X switch told tar to look for a file called backup.tar to tell it which files to exclude, which isn't what I meant to tell tar.

Let's return to the command that did work. To test that it didn't back up the file called apache2, I used grep to sort through tar's listing:

tar tf backup.tar | grep apache2

Since I just received my prompt back, I know my exclude file worked. It is interesting to note that since apache2 was really a subdirectory of www, all of the files in the apache2 subdirectory were also excluded from the backup. I then tested to see if the zope subdirectory was also excluded in the backup:

tar tf backup.tar | grep zope
www/zope-zpt/
www/zope-zpt/Makefile
www/zope-zpt/distinfo
www/zope-zpt/pkg-comment
<output snipped>

This time I got some information back, as there were other subdirectories that started with the term "zope," but the subdirectory that was just called zope was excluded from the backup.

Now that we know how to make backups, let's see how we can restore data from a backup. Remember from last week the difference between a relative and an absolute pathname, as this has an impact when you are restoring data. Relative pathnames are considered a good thing in a backup. Fortunately, the tar utility that comes with your FreeBSD system strips the leading slash, so it will always use a relative pathname -- unless you specifically overrride this default by using the P switch.

It's always a good idea to do a listing of the data in an archive before you try to restore it, especially if you receive a tar archive from someone else. You want to make sure that the listed files do not begin with "/", as that indicates an absolute pathname. I'll check the first few lines in my backup:

tar tf backup.tar | head
www/
www/mod_trigger/
www/mod_trigger/Makefile
www/mod_trigger/distinfo
www/mod_trigger/pkg-comment
www/mod_trigger/pkg-descr
www/mod_trigger/pkg-plist
www/Mosaic/
www/Mosaic/files/
www/Mosaic/files/patch-ai

None of these files begin with a "/", so I'll be able to restore this backup anywhere I would like. I'll practice a restore by making a directory I'll call testing, and then I'll restore the entire backup to that directory:

mkdir testing
cd testing
tar xvf ~test/backup.tar 

You'll note that I cd'ed into the directory to contain the restored files, then told tar to restore or extract the entire backup.tar file using the x switch. Once the restore was complete, I did a listing of the testing directory:

ls
www

I then did a listing of that new www directory and saw that I had successfully restored the entire www directory structure, including all of its subdirectories and files.

It's also possible to just restore a specific file from the archive. Let's say I only need to restore one file from the www/chimera directory. First, I'll need to know the name of the file, so I'll get a listing from tar and use grep to search for the files in the chimera subdirectory:

tar tf backup.tar | grep chimera
www/chimera/
www/chimera/files/
www/chimera/files/patch-aa
www/chimera/scripts/
www/chimera/scripts/configure
www/chimera/pkg-comment
www/chimera/Makefile
<snip>

I'd like to just restore the file www/chimera/Makefile, and I'd like to restore it to the home directory of the user named genisis. First, I'll cd to the directory to which I want that file restored, and then I'll tell tar just to restore that one file:

cd ~genisis
tar xvf ~test/backup.tar www/chimera/Makefile

You'll note some interesting things if you try this at home. When I did a listing of genisis' home directory, I didn't see a file called Makefile, but I did see a directory called www. This directory contained a subdirectory called chimera, which contained a file called Makefile. Remember, when you make an archive, you are including a directory structure, and when you restore from an archive, you recreate that directory structure.

You'll also note that the original ownership, permissions, and file creation time were also restored with that file:

ls -l ~genisis/www/chimera/Makefile
-rw-r--r--  1 test  wheel  406 May 11 09:52 www/chimera/Makefile

That should get you started with using the tar utility. In next week's article, I'll continue with some of the interesting options that can be used with tar, and then I'll introduce the cpio archiver.

Backup using tar command in linux

The tar program is an archiving program designed to store and extract files from an archive file known as a tarfile. A tarfile may be made on a tape drive; however, it is also common to write a tarfile to a normal file.

If you want to know more options about tar click here

Making backups with tar

A full backup can easily be made with tar:

# tar --create --file /dev/ftape /usr/src
tar: Removing leading / from absolute path names in the archive

The example above uses the GNU version of tar and its long option names. The traditional version of tar only understands single character options. The GNU version can also handle backups that don't fit on one tape or floppy, and also very long paths; not all traditional versions can do these things. (Linux only uses GNU tar.)

If your backup doesn't fit on one tape, you need to use the --multi-volume (-M) option:

# tar -cMf /dev/fd0H1440 /usr/src
tar: Removing leading / from absolute path names in the archive Prepare volume #2 for /dev/fd0H1440 and hit return:

Note that you should format the floppies before you begin the backup, or else use another window or virtual terminal and do it when tar asks for a new floppy.

After you've made a backup, you should check that it is OK, using the --compare (-d) option:

# tar --compare --verbose -f /dev/ftape
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/

Failing to check a backup means that you will not notice that your backups aren't working until after you've lost the original data.

An incremental backup can be done with tar using the --newer (-N) option:

# tar --create --newer '8 Sep 1995' --file /dev/ftape /usr/src --verbose
tar: Removing leading / from absolute path names in the archive
usr/src/
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/modules/
usr/src/linux-1.2.10-includes/include/asm-generic/
usr/src/linux-1.2.10-includes/include/asm-i386/
usr/src/linux-1.2.10-includes/include/asm-mips/
usr/src/linux-1.2.10-includes/include/asm-alpha/
usr/src/linux-1.2.10-includes/include/asm-m68k/
usr/src/linux-1.2.10-includes/include/asm-sparc/
usr/src/patch-1.2.11.gz

Unfortunately, tar can't notice when a file's inode information has changed, for example, that its permission bits have been changed, or when its name has been changed. This can be worked around using find and comparing current filesystem state with lists of files that have been previously backed up. Scripts and programs for doing this can be found on Linux ftp sites.

12.4.2. Restoring files with tar
The --extract (-x) option for tar extracts files:

# tar --extract --same-permissions --verbose --file /dev/fd0H1440
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/kernel.h

You also extract only specific files or directories (which includes all their files and subdirectories) by naming on the command line:

# tar xpvf /dev/fd0H1440
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/hdreg.h


Use the --list (-t) option, if you just want to see what files are on a backup volume:

# tar --list --file /dev/fd0H1440
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/kernel.h

Note that tar always reads the backup volume sequentially, so for large volumes it is rather slow. It is not possible, however, to use random access database techniques when using a tape drive or some other sequential medium.

tar doesn't handle deleted files properly. If you need to restore a filesystem from a full and an incremental backup, and you have deleted a file between the two backups, it will exist again after you have done the restore.

How-to Using Tar (Taring)

By SuperHornet from http://www.fluidgravity.com/ (http://www.fluidgravity.com/)

Ok well here is a short listing on how to use the command tar to backup you data..
Tar is solely an archiving app. Tar by its self wont compress files.

But you say "then what is a .tar.gz"

It’s a tar file that has been compressed with a different compression utility. The .gz=gzip is the compression app use to compress it.

Here is tar in its simplest form

tar -cvf filename.tar /path/to/files

-c means create
-f means filename (-f should always be last when you using syntax)
-v Verbose will display all the files its puts in the tar and error you might have incurred
You should see the filename.tar file in what ever directory you ran tar from.

You say "But I want to make the tarball compressed"

Well then -z is the option you want to include in your syntax

tar -zvcf filename.tar.gz /path/to/files


#notice I had to add the .gz extension.
-Z( no not -z) will run it thru the old compress app.

Now when I make a tarball I like to keep all the path's from which the file is in.
For this use the -P (absolute path)


tar -zPvcf filename.tar.gz /path/to/file


When I extract it I will see a new directory called /path
and under that I will see the "to" directory, and the "file" is under "to"

Now you say "I want to backup ALL my files in my home directory EXCEPT the temp directory I use". No problem.


tar -zPvcf myhomebackup.tar.gz --exclude /home/erik/temp /home/erik


The --exclude will give you this option, just slip it in between the tar filename and the path your going to backup. This will exclude the whole temp directory.

You say "Ok this tar thing is pretty cool but I want to backup only single files from all around the drive.

No problem, this requires a bit more work, but hey this is UNIX, get used to it.

Make a file called locations (call it anything you like). In locations place the full path to each file you want to backup on a new line. Please be aware that you have to have read rights to the files you are going to backup.

/etc/mail/sendmail.cf
/usr/local/apache/conf/httpd.conf
/home/erik/scripts


Now with the -T option I can tell it to use the locations file.

tar -zPvcf backup.tar.gz -T locations



Now if you want to backup the whole drive. Then you will have to exclude lots of files like /var/log/* and /usr/local/named/*

Using the -X option you can create an exclude file just like the locations file.



tar -zPvcf fullbackup.tar.gz -X /path/to/excludefile -T /path/to/locationsfile



Now a month has gone by and you need to update your myhomebackup.tar.gz with new or changed files.

This requires a extra step (quit your bitching I already told you why)
You have to uncompress it first but not untar it.



gunzip /path/to/myhomebackup.tar.gz


This will leave your myhomebackup.tar.gz mising the .gz.
Now we can update your tarball with -u and then we are going to compress it again.



tar -Puvf myhomebackup.tar /home/erik | gzip mybackup.tar


It will add the .gz for you.

Tar is a pretty old app and has lots of Fetchers.
I suggest reading the man pages to get a lits of all the options.


I have included a little perl script that I made so I can run it as cron job evernight and get a full backup each time.
It wouldn't be that hard to update the tarball but I just like full backups.
Feel free to use it.

If you want to extract the tarball that is compressed



tar -zxvf filename.tar.gz


-x extract

If it is not compressed then



tar -xvf filename.tar




#!/usr/bin/perl
#sysbkup.pl
#Created by Erik Mathis hornet@fluidgravity.com 7/02

#Change These paths to fix your needs.
my $filename="/home/sysbkup/backup";
my $exclude="/home/erik/exclude";
my $data="/home/erik/locations";
my $tar="\.tar";
my $gz="\.gz";


$file=$filename.$tar.$gz;

system ("tar -Pzcvf $file -X $exclude -T $data");

Re tar minor POSIX incompliance

• To: eggert@twinsun.com, lsb-test@lists.debian.org
• Subject: Re: tar: minor POSIX incompliance
• From: Joerg Schilling <schilling@fokus.gmd.de>
• Date: Tue, 26 Mar 2002 14:25:15 +0100 (CET)
• List-help: <mailto:lsb-test-request@lists.linuxbase.org?subject=help>
• List-post: <mailto:lsb-test@lists.linuxbase.org>
• List-subscribe: <mailto:lsb-test-request@lists.linuxbase.org?subject=subscribe>
• List-unsubscribe: <mailto:lsb-test-request@lists.linuxbase.org?subject=unsubscribe>

>From: Paul Eggert <eggert@twinsun.com>

>> From: Joey Hess <joey@kitenet.net>
>> Date: Mon, 25 Mar 2002 14:57:20 -0500
>> 
>> According to the test suite documentation, POSIX 10.1.1-12(A) says
>> that Fields mode, uid, gid, size, mtime, chksum, devmajor and
>> devminor are leading zero-filled octal numbers in ASCII and are
>> terminated by one or more space or null characters.

>OK, I'll change the behavior of GNU "tar" in a future release.

I am not sure what the text from Joey Hess should be related to...
... his mail did not reach this group.

>From looking at the archives created by GNUtar, I see the following deviations:

-	Checksum field repeats a bug found in ancient TAR implementaions.
	This seems to be a rudiment from early tests done by John Gilmore
	in PD tar where he did try to run "cmp" on PD-tar vs. Sun-tar
	archives.

	This is a minor deviation and easy to fix.

-	The devmajor/devminor fields are missing if the file is not
	a block/char device - here we see non left zero filled fields.

	A minor deviation that is easy to fix.

-	The Magic Version field contains spaces instead of "00".

	This is just a proof that GNUtar is not POSIX.1-1990 compliant
	and should not be changed before GNUtar has been validated to
	create POSIX.1 compliant archives.

...

>conformance by running the "tar" command.  A POSIX test suite should
>invoke the "pax" command instead.

While this is the correct answer from theory, you should take into account
that "pax" has not been accepted by a major number of people in the community.

AFAIK, LSB intends to be UNIX-98 compliant, so it would make sense to support
cpio/pax/tar in a way compliant to the SUSv2 document.

Let me comment on the current Linux status. We have:

-	GNUcpio which is neither POSIX.1-1990 compliant nor does it allow
	to archive files >= 2 GB.

	For a list of problems look into:

		ftp://ftp.fokus.gmd.de/pub/unix/star/README.otherbugs

-	GNUtar which is not POSIX compliant too but supports files >= 2 GB.

	Problems with archive exchange with POSIX compliant platforms:

	-	does not handle long filenames in a POSIX compliant way.
		This has become better with recent alpha releases, but
		gnutar -tvf archive still does not work at all.
		Archives containing long filenames and created with gtar
		cannot be read by POSIX (only) tar implementations correctly.

	-	Is for unknown reason unable to list archives created with other
		TAR implementations (e.g. Sun's tar on Solaris or star).
		For an example look into:

			ftp://ftp.fokus.gmd.de/pub/unix/star/testscripts/README.gtarfail

-	Pax (the version fixed by Thorsten Kukuk) is POSIX.1-1990 compliant
	but it is not able to handle files >= 2 GB.

as part of commercial Linux distributions. From a standpoint of what people 
might like to see, this could be better. A year 2002 POSIX OS should include at 
least one program that creates POSIX compliant tar archives _and_ supports 
large files.

People who get and compile software themselves may also use "star" which is 
POSIX.1-1990 andd POSIX.1-2001 compliant and supports files >= 2 GB.
So why is star missing from Linux distributions?


>Also, I should mention that GNU tar does not generate POSIX-format
>ustar archives, nor does it claim to.  Volunteers to fix this
>deficiency would be welcome, but that's a different topic.  It is a
>quality-of-implementation issue, and is not strictly a
>POSIX-conformance issue.

There is "star" which is POSIX compliant. A good idea would be to move
gnutar to /bin/gtar on Linux and put star on /bin/star and /bin/tar.
This way, Linux gets a POSIX compliant TAR and users of gnutar will be granted 
to have 100% backward compatibility when calling "gtar".

ftp://ftp.fokus.gmd.de/pub/unix/star/aplha/

If you don't like to do the transition too fast, here is an idea for an 
intermediate step:

Put star on /bin/star, install the star man page for "star" and "tar" and move
the GNUtar man page to "gtar".
/*--------------------------------------------------------------------------*/
Another topic:

>From a discussion at CeBIT, I am now aware of the fact that LSB did
"standardise" on the GNUtar options at:

	http://www.linuxbase.org/spec/gLSB/gLSB/tar.html

Let me comment on this too:

It seems to be a bad idea to standardize TAR options that are incompatible
with POSIX standards. So let me first introduce a list of incompatible options
found in GNUtar. The complete list is in:

	ftp://ftp.fokus.gmd.de/pub/unix/star/aplha/STARvsGNUTAR

/*--------------------------------------------------------------------------*/
Gnu tar options that (in the single char variant) are incompatible:

BsS	-F, --info-script=FILE		run script at end of each tape (implies -M)
s	-L, --tape-length=NUM		change tape after writing NUM x 1024 bytes
s	-M, --multi-volume		create/list/extract multi-volume archive
s	-O, --to-stdout			extract files to standard output
sS (+)	-P, --absolute-names		don't strip leading `/'s from file names
s	-S, --sparse			handle sparse files efficiently
s	-T, -I, --files-from=NAME	get names to extract or create from file NAME
s	-U, --unlink-first		remove each file prior to extracting over it
s	-V, --label=NAME		create archive with volume name NAME
s	-d, --diff, --compare		find differences between archive and file system
sP	-l, --one-file-system		stay in local file system when creating archive
sP	-o, --old-archive, --portability write a V7 format archive

B	Incompatible with BSD tar
s	Incompatible with star
S	Incompatible with Sun's/SVr4 tar
P	Incompatible with POSIX

+)	This option is the only option where star deviates from other tar 
	implementations, but as there is no other nice way to have an option to 
	specify that the last record should be partial and the star option -/ 
	is easy to remember as well as -P for Partial record is I see no need 
	to change star.

/*--------------------------------------------------------------------------*/

Please note that all these incompatibilities are "against" other TAR 
implementations that are much older than GNUtar. As as example, let me use the
-M (do not cross mount points) option in star which is available since 1985.

It looks inapropriate to me to include single char options from GNUtar that are not 
found in other tar implementations into something like LSB.

To avoid LSB systems to break POSIX.1-1990 and SVSv2, I would recommend to
change http://www.linuxbase.org/spec/gLSB/gLSB/tar.html so that the following
single char options will disappear (oder is the order from the web page):

-A	This option has low importance and there is no need to have a single
	char option for it.

-d	(*) Use by star with different semantic, the short option should not
	    1be in the LSB standard.

-F	(*) Used with a different semantic by BSD tar for a long time
	    the short option should not be in the LSB standard.

-G	The short option should not be in the LSB standard.

-g	The short option should not be in the LSB standard.

-K	The short option should not be in the LSB standard.

-l	This option violates the POSIX/SUSv2 semantics, it needs to be removed
	from the LSB standard.

-L	(*) The short option should not be in the LSB standard.

-M	(*) The short option should not be in the LSB standard.

-N	The short option should not be in the LSB standard.

-o	This option violates the POSIX/SUSv2 semantics, it needs to be removed
	from the LSB standard.

-O	(*) The short option should not be in the LSB standard.

-P	(*) The short option should not be in the LSB standard.

-R	The short option should not be in the LSB standard.

-s	The short option should not be in the LSB standard.

-S	(*) The short option should not be in the LSB standard.

-T	(*) The short option should not be in the LSB standard.

-V	(*) The short option should not be in the LSB standard.

-W	The short option should not be in the LSB standard.


*) Used by one or more other TAR implementations with different semantics
so defining it in LSB creates problems.

Jörg

 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
       schilling@fokus.gmd.de		(work) chars I am J"org Schilling
 URL:  http://www.fokus.gmd.de/usr/schilling   ftp://ftp.fokus.gmd.de/pub/unix


-- 
To UNSUBSCRIBE, email to lsb-test-request@lists.linuxbase.org
with subject of "unsubscribe". Trouble? Email listmaster@lists.linuxbase.org

Re Tar question

Sort of answered my own question. I downloaded and install star:

http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html

an enhanced version of tar that includes a name-modification option:

     -s replstr
          Modify file or archive member names named by a  pattern
          according  to the substitution expression replstr.  The
          format of replstr is:

               -s /old/new/[gp]


eks
 

Re Tar question -- star is recommended until GNU Tar 1.14 is

• To: LUCI List <luci-discuss@luci.org>
• Subject: Re: Tar question -- star is recommended until GNU Tar 1.14 is
• From: "Bryan J. Smith" <b.j.smith@ieee.org>
• Date: Sat, 07 Aug 2004 01:24:33 -0400
• In-Reply-To: <Pine.LNX.4.44.0408052156230.16610-100000@demeter.museum.state.il.us>
• Organization: Linux Users of Central Illinois
• References: <Pine.LNX.4.44.0408052156230.16610-100000@demeter.museum.state.il.us>
• Reply-To: luci-discuss@luci.org
• Sender: luci-discuss-owner@luci.org

---

On Thu, 2004-08-05 at 22:58, Erich Schroeder wrote:
> Sort of answered my own question. I downloaded and install star:
> http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html
> an enhanced version of tar

It's not really an "enhanced version of tar" but a more _POSIX_
compliant version.  That's why it has been a part of Fedora Core (FC)
since version 0.8* and _recommended_over_ GNU Tar 1.13.

Understand that cpio, tar and, their new replacement, pax, just write
what is known as "ustar" format.  The latest IEEE POSIX 2001 and X/Open
Single Unix Specification (SUS) 3 from the "Austin Group" defines a lot
of new functionality that really makes up for lack of capability in the
older 1988 and subsequent releases until the late '90s drafts.

This includes overcoming POSIX 1988+ path/naming limitations, as well as
newer POSIX 2001 capabilities like storing POSIX EA/ACLs.

In the meanwhile, the GNU maintainers decided to release their own
extensions that are not compliant.  It was a necessary evil, but now
that the POSIX/SUS standard has been updated, it's time for GNU to come
around.  The current GNU Tar 1.14 alpha adds these capabilities.

star actually had EA/ACLs support on Solaris** _before_ the POSIX
standardization, so adopting it for POSIX 2001 / SUS 3 ustar meta-data
format was easy.

Unfortunately POSIX 2001 / SUS 3 still does _not_ address the issue of
compression.  I hate the idea of block compressing the entire archive,
which renders it largely unrecoverable after a single byte error (at
least with LZ77/gzip or LZO/lzop -- BWT/bzip2 may be better at recovery
though).  That's my "beef" with the whole ustar format in general.

I would have really liked a flexible per-file compression meta-data tag
in the standard.  Until then, we have aging cpio replacements like afio.

-- Bryan

*NOTE:  This is the actual "disttag" versioning (i.e., technical
reasons) for pre-Fedora Core "community Linux" releases from Red Hat
that are now recommended for Fedora Legacy support (i.e., FC 0.8 is fka
"RHL" 8), in addition to any relevant trademark (i.e., non-technical)
considerations.

**NOTE:   legacy star used Sun's tar approach -- an ACL "attribute file"
preceding the data file, but using the same name.  That way if the tar
program extracting it was "Sun EA/ACL aware," it would read it, but if
not, it would just overwrite the attribute file with the actual when
extracted.  Quite ingenious of an approach.


-- 
Engineers scoff at me because I have IT certifications
 IT Pros scoff at me because I am a degreed engineer
    I see and understand both of their viewpoints
  Unfortunately Engineers and IT Pros only see in me
       what they dislike about the other trade
------------------------------------------------------
Bryan J. Smith                      b.j.smith@ieee.org

Lonestar announces new features

From: Press Release <press@lonestar.cactus.com>
Subject: LONE-TAR v4.0 Press Release
Date: Wed, 6 Aug 2003 17:25:13 GMT

FOR IMMEDIATE RELEASE:
======================
   LONE STAR SOFTWARE ANNOUNCES LONE-TAR(r) Version 4.0!
   MOUNT AIRY, MD (August 5, 2003) - Lone Star Software Corporation is
   pleased to announce LONE-TAR Version 4.0. complete backup and total
   disaster recovery software for UNIX(r) and Linux(r).  With the culmination
   of our latest version including such new features as license management,
   data encryption and bootable LONE-TAR backup and recovery media, we have
   set forth a future of unlimited possibilities.

NEW FEATURES in v4.0:
=====================
   o 256-bit encryption feature allows archived files to be encrypted on the
     backup media, keeping the data in them secure from unauthorized access.
   o Backups may be created on optical media, DVD-RAM, DVD+R/RW & CD-R/RW.
   o Introducing "BLT" - Bootable LONE-TAR backup with RESCUE-RANGER(r)
     disaster recovery combines a backup and recovery on one piece of media
     allowing a full system recovery to be a breeze.
   o OBDR(tm) [HP's One Button Disaster Recovery] Support - LONE-TAR and
     RESCUE-RANGER now support making your backups to tape bootable with
     HP Tape Drives that support OBDR.
   o Device Manager - You can go into the device manager at any time to
     change/add/remove a device, plus customize each one with its own unique
     settings such as compression and encryption.
   o Device Auto-Detection - Now LONE-TAR makes it even easier to configure
     your devices by detecting any recognized device on your system during
     install or anytime you add new devices.
   o License Manager - New licensing for demo and live products.  Demos can
     be extended or made live with an activation code without the need to
     reinstall.
   o LTX Menu Interface - GUI Menu Interface.  Now version 2.0 includes the
     same great features as the new redesigned character interface.
   o Character Menu Interface  - Redesigned and extremely easy to use. 
   o TAPE-TELL - better backup management.
   o Self-extracting installation wizard - makes new or upgrade installs
     a breeze, optionally preserving previous settings from prior versions.
   o Online Update - LONE-TAR can now be updated online, with an easy to use
     menu option. This new feature can check for updates to both LONE-TAR and
     RESCUE-RANGER with the touch of a button.
   o Plus many more script and program enhancements and improvements.

Availability:
=============
   o LONE-TAR v4.0  is available via our international network of value
     added resellers and distributors.  Quantity discounts, pricing
     incentives and competitive upgrades are available, as well as
     personal home editions and corporate site licenses.
   o Technical support is included for one full year, which includes
     24 x 7 disaster recovery support at no additional cost. 
   o Data Encryption feature is currently available at no additional cost.
   o Please contact our Sales office for additional pricing information at
     800-525-8649 or by email at sales@cactus.com.  Also, fully functional
     evaluation copies are available from the Lone Star Software Corporation
     web site at http://www.LONE-TAR.com.

About Lone Star Software Corporation:
=====================================
   Lone Star Software Corp. has been serving the UNIX/Linux community with
   leading edge backup solutions along with outstanding 24 x 7 technical
   support for over two decades.  For more information, visit us at
   http://www.LONE-TAR.com, contact Lone Star Software Sales at 800-525-8649,
   or by emailing us at sales@cactus.com.
 

[LUNI] Incremental Tar

Martin Maney maney at pobox.com
Sun Jun 8 21:25:09 CDT 2003

• Previous message: [LUNI] Incremental Tar
• Next message: [LUNI] Incremental Tar
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

On Sun, Jun 08, 2003 at 05:31:10PM -0500, Patrick R. White wrote: > So isn't this a good reason to use the dump/restore utilities to begin > with?

Maybe, but dump/restore is no panacea. Back in 1991, at LISA V, Elizabeth Zwicky of SRI presented a fascinating paper comparing the performance and problems of then-extant version of tar, cpio, pax, afio, as well as dump.

dump did well on most of the tests, but by its design it is capable of really frighetening errors if the filesystem is not quiesced (in practice, unmounted or mounted r/o would appear to be necessary) during dump. Also worth noting is that dump is quite filesystem-specific, and I seem to recall hearing that the ext2 version was interestingly broken a while ago. Since I don't employ dump, I can't tell you any more than that, sorry.

The only link I can find to the paper is from here:

http://www.phys.washington.edu/~belonis/

There's a postscript file and jpegs of the printed document. I thought there used to be something less cumbersome, but Google isn't finding it for me. It did find a good number of now-dead links, though. :-(

Ah, "zwicky backup torture" is a better search key. Still mostly passing mentions of this seminal work. Here's a more recent survey paper about *nix backup techniques:

http://citeseer.nj.nec.com/chervenak98protecting.html

Here's another useful compendium that seems to be currently maintained

http://www.cybertiggyr.com/gene/htdocs/unix_tape/unix_tape.html

OTOH, "cpio: Some Linux folks appear to use this" seems... odd.

Ah, google-diving!

--

A delicate balance is necessary between sticking with the things you know and can rely upon, and exploring things which have the potential to be better. Assuming that either of these strategies is the one true way is silly. -- Graydon Hoare

Recommended Links

• GNU tar 1.19 manual
• GNU tar - Tutorial
• Linux Journal: The Skinny on Backups and Data Recovery, Part 1 (Mar 04, 2000)
• Linux Journal: The Skinny on Backups and Data Recovery, Part 2 (Mar 14, 2000)
• Linux Journal: The Skinny on Backups and Data Recover, Part 3 (Mar 17, 2000)
• The Tao Of Backup: 1. Coverage

Reference

Solaris 9 tar manpage - Solaris tar understands ACLs, but GNU tar don't

AIX tar

GNU tar - Table of Contents

(gnu)tar - GNU version of tar archiving utility

Linux and Solaris ACLs - Backup

The Star tape archiver by Jörg Schilling, available at ftp://ftp.berlios.de/pub/star/, since version 1.4a07 supports backing up and restoring of POSIX Access Control Lists. For best results, it is recommended to use a recent star-1.5 version. Star is compatible with SUSv2 tar (UNIX-98 tar), understands the GNU tar archive extensions, and can generate pax archives.

download

GNU TAR

Debian Linux http://www.debian.org/
Mike Frysinger <vapier@gentoo.org>