The Linux Documentation Project: http://www.tldp.org – get the worldwide collection of HOWTOs.
Topic: Build Your Own Photo Server
Shell Commands:
Padraig Brady’s Command Line Tips
A Short Catalog of Bourne/Bash Shell Commands
Editor:
WINE and Emulators
Read Chapter 5, Accessing and Running Applications, Running Windows, DOS and Macintosh Applications, in the Red Hat Linux Bible.
Lots of software is available pre-compiled to run on Linux. Almost any UNIX-compatible program for which source code is available can be compiled, if packages aren’t already available. But what if you need to cross platforms?
- DOS
- Windows: older Win32
- Windows: NT and later
- Macintosh
Emulators “imitate” another environment. See http://rpmfind.net/ for an excellent resource for emulators (and a lot of other software). On that site, perform a search for these DOS-specific emulators:
- dosemu
- xdosemu
These packages are updated constantly; use the ones highest on the displayed lists. Keep in mind that you want the .i386 versions, not the .src versions (unless you want to compile them yourself). The dosemu package is more generally compatible; if you want to use xdosemu you’ll need to check your distro’s X library and glibc compiler for compatibility.
Download the .rpm package. Depending on system configuration, this file will arrive in your home directory or on your desktop if you’re running Gnome/KDE. The easiest way to perform this process is from the GUI; I’ll assume this is your working environment for this example.
Double-click the .rpm, and confirm all defaults as they’re offered.
When this process is complete (and it won’t give you much notice that it is), you can run:
whereis dosemu
If you installed as root, dosemu will be installed in /usr/bin/dosemu. Call it directly from that directory:
/usr/bin/dosemu
dosemu will be configured to run in /root/dosemu.
TAKE NOTE OF THE MESSAGE ABOUT SHUTTING DOWN DOSEMU:
YOU MUST COMMAND:
exitemu
TO LEAVE DOSEMU!
Now, let dosemu start. You can run:
dir
to see a listing of files in your “virtual” C: drive. Try:
edit config.sys
to see the familiar edit interface. Note that you have mouse support for the graphical menus.
Note that dosemu will often want access to your floppy drive. Normally only root and the floppy group can access it. Command:
chmod 666 /dev/fd0
to make it more widely available. Do be aware of the security considerations of doing this.
Be aware, this gets political: WINE truly is not an emulator. It is a Windows compatibility layer.
Go to http://winehq.org/ to learn more about WINE, particularly to check application compatibility. Follow the links to get pre-compiled packages for your distro.
Note that some packages are available in both i386 and i686 versions. Generally either will work for post-Pentium Pro-generation processors. See the WikiPedia entry for more discussion of i-numbering schemes.
You can get a commercial (and supported) version called WineX from Transgaming. If you’re specifically out to run Microsoft Office, you’ll need the commercial CrossOver Office package from CodeWeavers. But if your intent is to run games or mostly non-Microsoft applications for Windows, WINE by itself usually does the job.
You can install WINE from a Red Hat/Fedora GUI environment in the same way as dosemu: just double-click the .rpm and let it install.
Configuration files will be placed in /etc/wine/, and a pseudo-C: drive will be created in /usr/share/wine-c.
You should know the default drive mappings WINE is going to use:
|
Name
|
Drive Letter
|
Linux Directory
|
Device
|
| Floppy | A: | /mnt/floppy | /dev/fd0 |
| Hard Disk 1 | C: | /usr/share/wine-c | |
| CD-ROM | D: | /mnt/cdrom | /dev/cdrom |
| Temp Dir | E: | /tmp | |
| Home | F: | /home/username ( $HOME ) | |
| File system root | Z: | / |
This example assumes you have a connection to obtain two files: either a shared partition on a dual-boot machine, or a Windows share you can reach via Samba (smbclient), a floppy disk, a web site containing these files – any method available to you.
1. Get mspaint.exe.
2. Copy it to your /bin directory.
3. Run WINE. Command:
wine /bin/mspaint.exe
4. Note the message you receive.
5. Copy the DLL WINE tells you that you need. You’ll need it to be in the same place on your pseudo-C: drive that it would be under Windows: the C:/windows/system folder. Under WINE, this is at:
/usr/share/wine-c/windows/system
6. Now try to run Paint again.
There are two other emulation environments of which you should be aware.
The first is the mtools. Generally, these are DOS-like tools and commands that have the letter “m” added to the front. See Chapter 5 of the Red Hat Linux Bible for details. The important thing to know as you get started with mtools is that they operate as if your current working directory is the floppy drive (/def/fd0 or “A:”.) If you don’t have a floppy drive, or don’t have a disk in that drive, or your user doesn’t have permissions to access that drive, you’re in difficulty immediately. They’re worth exploring, but generally are useful only for dealing with a floppy disk you’re going to pass back and forth from Windows to Linux.
The second is ARDI. This is called an “executor” rather than an emulator, strictly speaking, because like WINE it is actually an application compatibility layer. NOTE that for most terminal-based UNIX software that’s compiled for Mac, you can get source code and compile that software for Linux. You’ll do the usual compilation dance (get used to it):
./config
make
make install
However, if you want to run Mac GUI apps, ARDI is your tool. See http://www.ardi.com/executor.php for extensive details, including lists of compatible applications.
grep
One of the most useful commands you will use is grep. Its syntax is:
grep [-b] [-c] [-E] [-F] [-h] [-i] [-l] [-n] [-s] [-v] [-w] search_text source_text
Think of this command as “get report.” The key is, report of what?
grep [-b] [-c] [-E] [-F] [-h] [-i] [-l] [-n] [-s] [-v] [-w] search_text source_text
|
-b
|
Precede each line by the block number on which it was found. This can be useful in locating a block |
|
-c
|
Print only a count of the lines that contain the pattern. |
|
-e
|
“Extended” grep, using regular expressions. Same as the egrep command. |
|
-f
|
“No regular expression” grep: returns results much more quickly because it does NOT interpret regular expressions. |
|
-h
|
Prevents the name of the file containing the matching line from being appended to that line. Used when searching multiple files. |
|
-i
|
Ignores case. |
|
-l
|
Displays the files that have the text, not the text within the file. |
|
-n
|
Precede each line by its line number in the file (first line is 1). |
|
-s
|
Suppress error messages about nonexistent or unreadable files. |
|
-v
|
Specifies you are looking for files that don’t contain the text. |
|
-w
|
Search for the expression as a word as if surrounded by \< and \>. |
|
string_to_match
|
The text that you are searching / not searching for. |
|
file_name
|
The file or files (or other input) to search. |
grep uses regular expressions to perform character matching or string matching. The characters include:
Matches a single character. Matches zero or more instances of the immediately preceding character. Example: C* if found would match C, CC or CCC … not to mention a blank string! Matches any character contained within the brackets. Represents the beginning of the line, so if you specified ^T grep would search for any line starting with a T. Represents the end of the line, so if you specified \.$ then grep would pull up any lines that ended with . The escape character: it means to take the next character literally, so you can search for characters like * that have special meanings: \* (See my Shell Scripting course page on Operators.)
grep: a caution about special characters
Note: Be careful using the characters $, *, [, ^, |, (, ), and \ in the pattern list because they are also meaningful to the shell. To be sure of your result, enclose the entire pattern list in single quotes (also called “literal quotes” or “hard quotes”): ‘… ‘.
grep ‘2 for $3’ specials.txt
grep will begin to make sense if you’re really clear about what goes in and what comes out of this command by default.
The basic syntax of grep is:
grep string_to_match input
grep searches the file, and the output is a printout of lines that contain matches for string_to_match.
What’s the most relevant difference between text files and binary files?
Text files are composed of lines.
In other words, at the end of each line there is a carriage return and a line feed (cr/lf). (Yes, there are other differences.)
Here’s the critical point:
Commands that process files will process them one line at a time.
|
End it right Do you know that Unix text-file line endings are different from Windows line endings? This is one really big reason why you shouldn’t use Word as your code editor. Try gedit, kedit or nedit in Linux, pico on an AIX server, or a Unix-compatible text editor like Arachnophilia in Windows. You should also see if your system includes the dos2unix utility. |
You could simply name a file:
grep chocolate myfile.txt
which will return any lines in the file myfile.txt that contain the string “chocolate.” What you get back is a printout, to the terminal screen, of the matching lines.
You don’t have to just let the matching lines print to the screen, though.
Use the pipe character | to redirect the results to another command, or use the redirect character > to write the results to a file. Use your imagination to see how useful this could be:
grep Denmark Hamlet.txt>term_paper_quotations.txt
grep will also accept standard input instead of a file, provided you don’t supply a file. Remember, though, that you will have to supply a standard input:
cat myfile.txt | grep chocolate
I like chocolate
chocolate cookies
Leaving the filename argument blank, as in the previous example, will work just fine a lot of the time.
It won’t work well at all if I’m trying to search the output of a cat command for the word Hamlet, for instance, and there happens to be a file named Hamlet. It’s too easy to become unclear whether Hamlet is the filename argument or the search string.
That’s why we use a dash to indicate standard input, just for clarity:
cat myfile.txt | grep chocolate –
I like chocolate
chocolate cookies
When you think about it, the fact that grep accepts standard input is pretty wonderful. How many ways can we generate standard input?
By reading a file:
cat myfile.txt | grep chocolate –
By listing files:
ls –la | grep txt –
By listing processes:
ps aux | grep tty –
By listing users:
who | grep studenth –
The fact that we can feed grep input from a variety of sources isn’t the only great thing about grep.
What’s even more wonderful is that the matching string doesn’t have to be just a simple string, like “chocolate.”
Now, consider this: $ is a special character within regular expressions; it means “end of line.”
So how do you search for lines containing the dollar sign without grep thinking you’re looking for a line ending?
You have to “escape” the special meaning of $:
\$
Try this test: first, create two files:
touch hello.gif hello1gif
Now do a long listing of your directory and pipe it to grep to look for hello.gif.
Go ahead, actually do it before reading on.
What command did you try ?
To search for a line containing the text hello.gif, the correct command is
ls | grep ‘hello\.gif’ –
since
ls | grep ‘hello.gif’ –
will match lines containing hello-gif , hello1gif , helloagif , etc.
Now try:
ls | grep hello\.gif
What results do you get? Why?
How do you look for a list of possible matches, not just one?
grep –e “support\|help\|windows” myfile.txt
Would search for “support,” “help” and “windows” in the file myfile.txt.
ANY of these phrases would produce a match.
Note that grep –e is the same as egrep: either command searches using regular expressions.
The grep “or” operator consists of these two characters together:
\|
What’s really happening here is that we are escaping the “regular” meaning of the pipe character, in favor of a “special” meaning. Note that you MUST enclose this inside single or double quotes.
grep “cat\|dog” myfile
matches lines containing the word “cat” or the word “dog.”
grep “I am a \(cat\|dog\)” myfile
matches lines containing the string “I am a cat” or the string “I am a dog”.
To match a selection of characters, use brackets: []
For example:
[HhJ]ello matches lines containing hello or Hello or Jello.
Ranges of characters are also permitted:
[0-3] is the same as [0123]
[a-k] is the same as [abcdefghijk]
[A-C] is the same as [ABC]
[A-Ca-k] is the same as [ABCabcdefghijk]
There are also some alternate forms :
[[:alpha:]] is the same as [a-zA-Z]
[[:upper:]] is the same as [A-Z]
[[:lower:]] is the same as [a-z]
[[:digit:]] is the same as [0-9]
[[:alnum:]] is the same as [0-9a-zA-Z]
[[:space:]] matches any white space
Trying to find lines that do NOT contain a string? Use the -v option:
grep -v Denmark Hamlet.txt
An expression consisting of a character followed by an escaped question mark matches one or zero instances of that character.
bugg\?y
matches:
bugy
buggy
but not
bugggy
Let’s say we want to find all references to Frederic or Fred, by making the string “eric” following “Fred” optional.
To do this we’ll make grep treat “eric” as a single letter.
An expression surrounded by “escaped” parentheses is treated like a single character:
Fred\(eric\)\? Smith
matches Fred Smith or Frederic Smith.
\(abc\)* matches abc , abcabcabc etc. (i.e. , any number of repetitions of the string abc , including the empty string)
Note that we have to be careful when our expressions contain white spaces or stars. When this happens, we need to enclose them in quotes so that the shell does not mis-interpret the command, because the shell will parse whitespace-separated strings as multiple arguments, and will expand an unquoted * to a glob pattern.
So to use our example above, we would need to type:
grep “Fred\(eric\)\? Smith” filename
You can use escaped braces \{ \} to indicate a count of matches for the preceding item.
A good example is phone numbers. You could search for a 7 digit phone number like this:
grep “[[:digit:]]\{3\}[ -]\?[[:digit:]]\{4\}” filename
This matches phone numbers, possibly containing a dash or whitespace in the middle.
And all this is only the beginning…
grep matches strings in lines of text.
All text files consist of lines of text (with line returns at the end).
You can use wildcards, lists, or patterns to match.
If you need something fancy, find and modify an example!
grep man page: http://linuxcommand.org/man_pages/grep1.html
GNU grep manual: https://www.gnu.org/software/grep/manual/grep.html
GNU grep texinfo: http://www.gnu.org/software/grep/doc/
SS64’s grep page: http://www.ss64.com/bash/grep.html
The strings command searches a binary file for text strings, primarily so you can figure out what it does:
strings /bin/bash | less
If you’re a real propeller-head, doing core-dump debugging for instance, you may use the od command. This command shows the contents of a binary file in octal (base 8) format:
od /bin/bash | less
For a real treat, render the file in hexadecimal (base 16) format:
od -x /bin/bash | less
Text Editors
First, about vi
The program called by the command vi on Linux systems is actually vim, or “vi improved.”
On other Unix systems, for instance an AIX system, you may be using either vi or vim (and won’t be able to tell).
To find out about vi/vim, on the command line type:
vi
Type :q to exit.
vimtutor
At the command line, type:
vimtutor
This gives you a nice tutorial you can review if you’re interested in using vi frequently.
The basics of working in vi
To open vi, just type:
vi
If you want to edit an existing document, type:
vi filename
You start in Command Mode. You can’t edit in this mode, just do things like save and quit. To get into Insert Mode, where you can edit text, issue the single-letter command:
i
Now you’re typing. Basic navigation is easiest with the arrows on your keyboard. Move around, type and delete just like you always would. But sooner or later you’ll have to leave Insert Mode. To do this, press the
<escape>
key. You’re back in Command Mode.
Need help? Type:
:help
To simply leave, type:
:q
That’s a colon and the letter “q”. If you’ve changed text, vi will complain that you haven’t saved. To quit without saving:
:q!
The exclamation point forces the issue. Or you can choose to write your changes (i.e. save them):
:wq
If you opened vi without specifying a file name, you’ll need to supply a file name to save:
:w filename
or
:wq filename
if you’re writing and quitting.
The SourceForge Beginner’s Guide to Vim: an excelent, extensive introduction with lots of examples
A basic vi reference sheet:
The Vi Cheat Sheet at http://www.lagmonster.org/docs/vi.html
For advanced reference details:
See the Advanced Vi Cheat Sheet at http://www.lagmonster.org/docs/vi2.html.
And for really detailed information:
Jerry Peek’s The Joy of Vim series:
Part One, at http://www.linux-mag.com/id/2996/ has particularly good info on vim’s help system, and windowing and tabs in vim
Part Two, at http://www.linux-mag.com/id/2991/
Part Three, at http://www.linux-mag.com/id/3028/
Version 7 of Vim offers many new features:
See William Nagel’s “Vimming to New Heights” at
http://www.linux-mag.com/id/2906/
pico
If you find vi frustrating or just plain silly, you can use another text editor on UNM systems. (It’s not included in every Linux distribution, for instance the Fedora we’re using in this class.)
Command:
pico
pico, unlike vi, allows you to begin typing immediately. You’ll see a menu along the bottom of the terminal, which makes life much nicer.
What’s the command to save a file?
What’s the command to leave pico?
mcedit, nedit
Other terminal-based editors you may find on your system are mcedit (the Midnight Commander Editor) and nedit. Try:
mcedit filename
If you have mcedit, you’ll see a screen with function-key operations listed across the bottom of the screen.
nedit filename
will give you a menu-based editor.
gedit, kedit, nedit
If you have a graphical environment available, the next step up in text editors is XWindows-based text editors like gedit, kedit, or nedit.
All of these are clean, simple editors along the line of Microsoft WordPad. You’ll find the menus and command buttons very familiar.
Best of all, gedit is easy to invoke from a command prompt:
gedit filename
emacs
In the Unix world, emacs is the grand old text editor. Some people have accused emacs fanatics of trying to create a complete operating system from their word processor.
emacs operates in modes: text mode, Perl mode, C mode, and so forth.
You can open emacs with the simple command:
emacs
From here on, nothing is simple. emacs requires that you, the user, know a number of keyboard functions.
| Some Common emacs Keyboard Functions | |
| Key | Purpose |
| Ctl-a | Clear your terminal’s screen |
| Ctl-e | Reset your terminal to default settings |
| Ctl-h | Display a user’s information |
| Ctl-d | List all currently logged-in user |
| Ctl-k | List current users and their tasks |
| Esc-d | Show your own current logged-in name |
| Ctl-x + Ctl-c | Lists your UID and GIDs |
| Ctl-x + Ctl-s | Display today’s date |
| Ctl-x + Ctl-w | Display this month’s calendar |
| Ctl-x + u | Leave a sub-shell |
Your system may have xemacs installed. On a Red Hat-based system, you can check by commanding:
rpm -q xemacs
This is an XWindows-based version of emacs that can give you some “training wheels.”
Finally, Arachnophilia and other Unix-compatible editors
There’s one very important point to keep in mind:
Line endings in Unix are different from line endings in Windows.
Thus it’s very important which text editor you use in Windows when you’re scripting for Unix.
Enter Arachnophilia.
On the home page for this class you’ll find a link to the Arachnophilia download page. The current version is written in Java. I personally like an older version, written in Visual Basic.
Download either or both and try them out. Find the menu item that deals with line endings!
And remember, Arachnophilia ain’t the only game in town for crossing the Unix/Windows border….
Converting Windows Files
If you find you’ve uploaded (or been sent) a Windows-based text file, you always have a quick fix:
dos2unix file1 file2 …
Be sure to check out man dos2unix for some interesting options.
Resources
Unix Manuals: vi Reference: http://www.unix-manuals.com/refs/vi-ref/vi-ref.htm
Building Shell Commands
Many functions in Unix deal with text strings like file names. You’ll need at least two file name wild cards (or shell wild cards) to make life easier.
The first wild card is “ * “, which means “match any character or string of characters.” It’s handy, for instance, for finding a file:
ls -l chap*
will get you a listing of all the files (in your current directory) with names beginning with “chap”.
It will also attempt to return listings from any subdirectory with a name beginning with “chap”.
The second wild card character to know is “?”. It matches only a single letter.
ls chapter?
will return any file names with only a single letter after chapter.
Variables: symbolic names used to represent values stored in memory
There are three types of variables:
Configuration variables
Environment variables
Shell variables
Store information about the setup of the operating system
Cannot be changed
HOME
LOGNAME
TZ (Time Zone)
Determine the characteristics of your session
Can be changed by you or your sys admin
PATH
PS1 (Shell prompt)
PWD (Current Working Directory)
Created at the command line or in a shell script
Useful in shell scripts to store temporary information
dog
memo
my_var
today
~$: NAME=Becky
The string Becky is assigned to the shell variable NAME.
To display the value of NAME, it must be preceded with a dollar sign ($), for example
~$: echo NAME
displays NAME.
~$: echo $NAME
Displays Becky.
Assigning commands to variables
The backquote (`) operator can be used to store the output of a command in a variable:
~$: TODAY=`date`
~$: echo $TODAY
Displays today’s date.
~$: TODAY=date
~$: echo $TODAY
Displays the word “date.”
Recall the pipe special character, and its use in feeding the stdout of one command into the stdin of another command:
ls –la /bin/ | less
Any command that can accept Standard Input and produce Standard Output is called a filter command.
| Common filter commands | |
| Command | Description |
| sort sort -r |
Sorts lines in a file alphabetically
Reverse-alpha sorts lines in a file |
| wc wc -l wc -w wc -c |
Counts number of lines, words and characters in a file.
Counts number of lines Counts number of words Counts number of chars |
| pr pr -d |
Formats a file for printing, w/ date and page number.
Formats for double-spaced printing. |
| tr |
“Translate” or “transform”
Used to replace characters in a text file. |
| grep |
Displays those lines in a given file that match a string of your choice.
|
| nl | Prefixes a line number to the lines in a file. |
| awk | Manipulates text using pattern/action expressions. |
| sed | Manipulates text using search/replace expressions. |
The redirection character ( > )
Also recall the redirection character, which sends the output of a command to a text file:
(users) > /var/users.log
If you want to append your command output to a file, rather than “clobbering” it, use the append character:
(users)>>/var/users.log
Sometimes you need more than one command to do the job. Use the semicolon:
(date;users)>>/var/users.log
Running a command in the background ( & )
To execute a command in the background, follow the command with an &. When you run a command that you know is going to take a while to complete, running it in the background lets you continue working in the mean time. On many Linux distros, backgrounding a command gives it a nice value of 4 (see the nice section). Compare:
ls –laR /
with
ls –laR / >/files.txt &
The latter executes ls in the background and immediately gives you a new prompt to continue working.
Use the command:
jobs
to view background job IDs.
Add a ” % ” to kill a background job:
kill %1
Move a backgrounded job to the foreground:
fg %1
Move a foreground job to the background:
Ctrl-z #to assign a background ID to the job
bg %1 #to background that job; Use the ID supplied by the previous command
In Unix, an argument isn’t something you get into with your sibling.
An argument is a parameter, like specifying which directory to list when you give an ls command:
ls –la /home/glenn
In this case, /home/glenn is the argument.
Expanding command output as an argument
Consider this chain of actions.
I want to “touch” every file in my home directory, so I could issue the commands:
touch /home/glenn/* ; touch home/glenn/ *.*; touch /home/glenn/.*
This will refresh the last-modified date of every file in my home. (Now, why would I want to do this?)
But, I decide I don’t want to touch all my files; only the .txt files. So instead I command:
find /home/glenn | grep \.txt
to find my .txt files.
Now I have the list of files I want, I can use that list (which is really the output of my last command) as the input for the touch command:
touch $(find /home/glenn | grep \.txt)
In this case, I’m indicating that I want to run the find command first by bracketing it with:
$( my_command )
But I could also use the format:
$‘my_command’
You can also bracket and expand math expressions, using slightly different notation:
echo “2 plus 2 equals $[2+2].”
Finding Files
To use the locate command, you will need to have an slocate database set up on your system. On many systems it is updated periodically by the cron daemon. Try the locate command to see if it will work on your system:
locate hello
This should list every filename on your system that contains the string “hello.” If this doesn’t work, you can use the commands:
slocate -u
or
updatedb
to build the slocate database. This may take a few minutes. On most systems with slocate installed, slocate is run periodically by the cron facility.
find
The find command lets you search for files by criteria, for instance by user, size or date. The basic syntax is:
find directory_to_start_from [options_if_any] string_to_find
By default, the string_to_find is the file name, or some part of the file name. But you can search by criteria too, such as the owner of the file:
find / –user studenth
In a similar way, the -group option restricts searches to files owned by a group. Or search by the size of the file:
find / -size 100k
Another truly helpful option, if you have remote mounts on the workstation, is -mount:
find / -mount file_name
This restricts find to local disks only.
Note that a bare “find” command lists every file on the workstation.
This command only works for executables, and it will find their man pages and (particularly useful) their configuration files as well. For instance:
whereis bash
will return the full path to bash, bashrc and the man page bash.1.gz. This is handy for finding the true paths to programs that aren’t in your PATH, for instance.
which finds an executable file. Unlike whereis, it does not find man pages as well.
which bash
This command will show the full path to the bash executable.
Get a manual page for a command:
man command
man tracert
What? There’s no output? You need to build the man page database:
makewhatis
Search by keyword:
man –k keyword
man –k columns
apropos columns
(apropos gives the exact same output as man -k.)
Info pages:
info command
info coreutils utility_name
where utility_name is a utility like ls.
And finally, yes, there is a help command, not available for all commands but sometimes useful:
help echo
About those numbers following the command in man page output:
For instance WHOAMI(1).
| Manual Page Sections | ||
| Section | ||
| 1 | Commands available to all users | |
| 2 | System calls | |
| 3 | Library routines | |
| 4 | Special device files | |
| 5 | File formats | |
| 6 | Games | |
| 7 | Macro packages | |
| 8 | Commands available only to root | |
| 9 | Kernel routines | |
| n | New: not yet categorized | |
LinuxCommand.org: http://linuxcommand.org/
A good intro to commands and scripting.
SS64: Unix, Oracle, Windows and Mac commands: http://ss64.com/
A great site devoted solely to the ins and outs of the command line in various environments.
Compression Utilities
tar and zip
The zip utility has been around since long before the advent of WinZip. In fact it predates pkzip, the first popular DOS compression/archiving utility.
Unix, as usual, breaks things down to more elemental processes: zipping (compressing a file) is separate from archiving (enclosing a group of files in an “envelope”).
In the Linux world, there are several zip programs: gzip (GNU zip), bzip, zip and compress.
tar is used to pack the entire contents of a directory or directories into a single file called a tarball, which can then be backed up to tape or saved as a file.
tar preserves the entire directory organization including file ownership, permissions, links, and the directory structure.
Note that tar cannot archive files or directories with names longer than 255 characters.
tar functions
The most commonly used tar functions are:
c – create an archive
x – extract files from an archive
t – show the “table of contents” of an archive
(you must use -z as well, if the archive is compressed)A – append one archive to another
r – append files to an existing archive
P – store files with absolute pathnames
u – appends files to an archive if they are newer than the same files already in the archive
W – verify contents of the archive after creation
tar options
Additionally, there are commonly used options:
v – verbose
f <filename> – use the specified file
z – gzip/gunzip
–gzip – gzip
–gunzip – unzip
j – bzip/bunzip
Z – use the compress utility
gzip: Lempel-Ziv compression (LZ77)
The gzip utility compresses files like WinZip or pkzip. It can achieve up to 60-70% compression, though often this is much less. The command itself is used very simply:
gzip zipped_file.gz file_to_zip.txt
gzip -d zipped_file.gz #d for “decompress”
gzip -r zipped_file.gz /directory/to/zip
gunzip zipped_file.gz
zcat zipped_file.gz
zmore zipped_file.gz
The tricky part comes with the huge number of options. Particularly, you can control the level of compression, on a scale of 1 (fast) to 9 (small):
gzip -v -1 zipped_file.gz file_to_zip.txt
bzip: Burrows-Wheeler Block Sorting Huffman Coding Algorithm
bzip can’t compress a directory full of files, while other algorithms can. You also can’t zcat ar zmore a .bz file. But you often get higher compression: 50-75%.
bzip2 -v file1 file2
bzcat file1.bz2
bzmore file1.bz2
bzless file1.bz2
bunzip2 -v file1.bz2 file2.bz2
bzip -d file1.bz2
bzip -f file1.bz2 #force
bzip -k file1.txt #keep the original file; otherwise deleted
bzip -q file1.txt #quiet
Unpacking files
When you download a file, you often get an archive that contains several – perhaps many – files within it. Archive files can also contain directories, which can contain files and subdirectories. Generally, these archive files have names ending in .tar, .tgz, .tar.gz, tar.bz2 or .zip.
To get at the files contained in the archives, you must unpack them.
If the file is a .tar file, open a terminal window, move to the directory containing the .tar file, and issue the command:
tar xvf filename.tar
The command will unpack the contained files to your current directory. You can use the ls command to view the results of your work.
(Quick: what do the three options mean?)
Unpacking .tgz and .tar.gz files
If the packed file is a .tgz or .tar.gz file, it has been archived and then compressed. So, the previous command, which merely unarchives the file, won’t work. To unpack a .tgz or .tar.gz file, open a terminal window, move to the directory containing the .tgz or .tar.gz file, and type:
tar zxvf <filename>
The command will expand the compressed file and then unpack the archived files to your current directory.
To unpack a .bz2 (bzip) file, open a terminal window, move to the directory containing the . bz2 file, and type:
tar jxfv file.tar.bz2
Notice that the only difference is the “j” option replacing the “z” option.
If the packed file is a .zip file, you shouldn’t issue a tar command at all. Instead, open a terminal window, move to the directory containing the compressed file, and type:
unzip filename.zip
Like the previous command, this command also unpacks the contained files to your current directory.
Archiving files: Making a tarball
Why do we do this? For one, it’s easier to send someone a single file than a whole mess of them. But for another, it’s often important to send files with permissions and file attributes intact, something that never happens with most forms of copying.
To create a .tar file, open a terminal window, move to the directory containing the files or directories you want to pack, and type:
tar cvf mynewfile.tar file1 file2 …
You can list as many files and directories as you want; just separate each entry from the previous entry with a space.
If you want to compress the packed file, simply include the z flag on the command line:
tar zcvf mynewfile.tgz file1 file2 …
Be very aware that tar syntax gets quite tricky. Resort to man tar often, and/or follow instructions diligently.
A potential advantage of a .zip file is that familiar Windows programs use the .zip format. To create a .zip file, at a command line and in the directory containing the files you want to zip, type:
zip myfile.zip file1 file2 …
where myfile.zip is the name of the .zip file you want to create and file1 and file2 are files or directories you want to pack. As in the .tar command, the ellipses (…) indicate that your list can go on forever.
The compress utility: LZW compression
compress -v file1 file2
compress -f smallfile1 symlink1
uncompress file1.Z file2.Z
zcat file1.Z
zmore file1.Z
zless file1.Z
The area of archives is a great place to look at how Unix commands can be assembled.
Imagine almost any scenario in which you wanted to zip up a group of files. On the command line, the most heinous chore is listing those files.
How about using the find command to do the listing for you?
find . –name “*.[ch]” –print | zip newfile.zip -@
Which means:
“Find, in this very directory ( . ), the files with names like anything dot c or h, print that list, and pipe it to zip, which will create a new zip file, and put the files listed by find into it.”
Check out man zip to find out about using -@.
find . –name “*.[ch]” –print | zip new.zip -@
Notice that text string in weak quotes? The square brackets are part of a whole other phenomenon in Unix: Regular Expressions.
Suffice it to say for now that in this case, anything.c or anything.h would match the expression. Just be aware that these things exist and that the letters inside the brackets describe a list or range of matches.
Want to find files, tar them up, maybe even remove the originals once you’ve created your tarball? Consider this:
find * -type f -mtime +90 -print > list.txt
tar -create -gzip -file my.tar.gz -files-from list.txt -remove-files
Mounting & Dismounting
Mounting is simply the process of “plugging” a directory into your file system tree. You can mount a whole partition, for instance when you add a hard drive. Or you can mount a shared network directory into your local file system tree.
There are a couple of critical principles to know. First, you will use a directory that already exists on your file system tree as a point to which you’ll mount. Normally you’d create an empty directory for just this purpose, but in the event that you use a directory that already has contents, those contents won’t be available until you dismount what you’ve mounted over them.
Second, file system type matters. Your system may or may not have support for various file systems, and in the case of some file systems like NTFS, you may have read-only access. You’ll be expected to know, in most cases, what type of file system you’re mounting as well.
mount is, more or less, a simple command. Its basic syntax is:
mount [options] device_to_mount mount_point
For example, to mount a partition to a directory:
mount -t ext3 /dev/hdc5 /projects
This assumes that you’ve created a directory, /projects, where you’ll access the contents of the new hard drive.
Use the full and formal:
mount -t ext2 /dev/fd0 /mnt/floppy
If the floppy disk is already listed in /etc/fstab (it ought to be) the mount command can consult that file if you supply only the barest information:
mount /dev/fd0
or
mount /mnt/floppy
CD-ROMs generally (once they’re created) use the iso9660 filesystem, and are (generally) read-only:
mount -r -t iso9660 /dev/cdrom /mnt/cdrom
Command:
mount -t ext3 /dev/hdc1 /newdirectory
ss
mount -a
The File System Table (fstab) file holds a list of mountable drives. Not everything in this list mounts at boot; some entries are there to allow easy mounting of removable drives, for instance.
This the fstab file has two roles:
- It mounts partitions at boot time, and
- it holds information that allows users to mount partitions/drives with minimal information.
The file has six columns or fields:
- The device to mount, e.g. /dev/hda1
- The mount point, e.g. /home
- The filesystem type – ext3, swap, proc, udf, auto, etc.
- Mount options – defaults, ro, rw, noauto, owner, GID and more.
- dump# – 1=Back it up, 2=No backup
- fsck# – 0=No check, 1=Check first, 2=Check later
Filesystems with the noauto mount option are not mounted at boot time.
Simply command:
mount
and you’ll get a list of currently mounted filesystems. Note that this command abstracts this information from the file /etc/mtab, which you can view directly yourself:
cat /etc/mtab
This can be a critical question when you’re performing an unmount. Command:
fuser -u /mnt/floppy
The -u option specifically checks for users. You can kill the processes accessing files on the filesystem with a -k:
fuser -k /mnt/floppy
Or be nice and ask first:
fuser -ik /mnt/floppy
Use the umount command, with either the device file or the mount point as the argument:
umount /mnt/floppy
umount /newdirectory
Note that floppies, CDs and other removable media must be unmounted before they are removed.
LinuxPlanet: Adding Additional Hard Drives In Linux, at http://www.linuxplanet.com/linuxplanet/tutorials/4232/1/
Sharing Files
A Safe Environment for Multiple Users
Unix systems have multiple users – and a permissions system that lets each user share or protect their files.
This unique system has some odd twists. For example, did you know that, to rename or remove a file, you need write permission for the directory that the file is in?
Users and Groups
In Unix, every user has a username (like zoe) and a corresponding user ID number (or UID) like 1324. Every user also belongs to one or more groups, where each group also has its own name and ID number (the GID). You can run the id command to see your own user and group information:
id
uid=1324(zoe) gid=1324(zoe) groups=1324(zoe),111(staff)
Home Owners
Every file and directory has a user owner and a group owner, as well as a set of permissions that dictates what access is allowed for the file’s user, group, and others. Use pwd to check your location, and ls –dl to get information about the directory itself:
pwd /home/zoe
ls –dl
drwxr-x–x 5 zoe zoe 4096 Nov 2 10:33
A quick review of ls
The first character on each ls -l output line shows the file’s type. Here, d is a directory file. This file’s user owner is zoe, and its group owner is the group named zoe.
The next characters are the types of access allowed for the user owner (here, rwx), for members of the group (r-x), and for all others.
The meaning of the rwx permissions for files is simple: r permits reading from a file, w lets you modify it, and x lets you execute it (as a program). However, how rwx applies to directories isn’t so obvious.
Directory magic
Read (r) access to a directory lets you list its contents using ls or with a GUI file browser.
A directory is a special type of file that contains (among other things) the names of files that are stored “in” the directory. So, being able to read a directory file lets you see the names stored in it.
Writing to directories
In the same way, write (w) access to a directory lets you modify the directory’s contents – to change the names of the files stored in that directory.
To add, remove, or rename a file, you need write permission on the directory that contains the file’s name. If a directory is write-protected, the list of entries in that directory cannot be changed – accidentally or on purpose.
Directory x permission
Execute (x) permission on a directory lets you access entries in that directory. Even if you have read permission on a directory (which lets you see what’s in it), you can’t use any of its entries unless you also have x permission.
For example, even if an individual file’s access permissions allow you to read and write to a file, you still can’t read or write the file unless you have execute permission on its directory.
Strict parents
In fact, you can’t access a directory unless you also have execute permission on all of its parent directories, all the way up to the root directory! Similarly, without execute permission on a directory, you can’t browse its subdirectories.
But you don’t need read (r) permission on a directory to access an entry in it if you know the entry’s exact name.
Catch-22: Graphical Interfaces
Graphical applications may not handle Unix permissions very well. Because they usually show dialogs with icons for the files in a directory, they have to be able to read the directory. So, if users try to reach zoe’s xfotos subdirectory by clicking through /home to /home/zoe, they’ll get some sort of error on her home directory because the GUI app can’t read it.
Can’t get it write
Some applications – graphical and otherwise – also aren’t sophisticated about write permission: they assume that, if they can read a directory, they also can write to it. Writes can fail in strange ways because, although a particular file in a directory may be writable, the directory itself isn’t.
Bypassing the unreadable
GUI file-open dialogs will generally let you type (or paste) an absolute pathname (like /home/zoe/xfotos) into their “location” box.
This lets you bypass unreadable intermediate directories.
Granting Permission
Unix permissions let groups of users share files – even modify or remove other users’ files – if they’re given permission. Directory and file access permissions are often set to allow general file sharing, so let’s see a specialized example.
Let’s say that Zoe’s home directory is on a networked filesystem shared between all computers in her office. She could tell her friends to point their photo viewers at her directory, ~zoe/xfotos.
The shell expands ~zoe to the absolute pathname of zoe’s home directory; Zoe herself can use just ~.
Just Enough Slack
Her permissions give everyone just enough access to read the photos, but not enough access to add, delete, or modify files:
ls -dl ~/ ~/xfotos
drwxr-x–x zoe zoe…/home/zoe
drwxr-xr-x zoe zoe…/home/zoe/xfotos
ls -l ~/xfotos
-rw-r–r– zoe zoe … alex.jpg
-rw-r–r– zoe zoe … boss.jpg
Easy Sharing
This example shows one of the strengths of a Unix filesystem: it’s easy to share when you want to. Zoe didn’t need to fill friends’ mailboxes with individual copies of her photos, or configure the office Web server so her friends (but no one else) could see those files. Instead, she simply “opened” her directory and her files just enough to let people read them.
Private Areas
An easy way to keep your private files private is to make specific directories in your home directory with owner-only access. For instance, make a directory named personal and type:
chmod 700 personal
to give yourself all (rwx) permissions, but no access to members of your group or to others. All files and subdirectories under personal – no matter what their permissions – are protected from everyone except the system superuser.
Give your private directories a less obvious name than personal or private, which could be a red flag for crackers and others who want to get into your account somehow.
Default Permissions: umask
If you’re concerned about security but also need to share files, check the permissions of every file – including “hidden” files like .bashrc – before you start sharing.
You should also set the correct umask. You’ll recall from the previous unit that the umask defines the default permissions of new files and directories. The umask is a bit mask that specifies which permissions to deny when creating a file. You can see the current umask setting by typing umask at a shell prompt. To set it, use umask nnn, where nnn is the new umask.
Groups
Recall the output of id at the start of this presentation. zoe belongs to two groups: zoe and staff. Unix groups can have many members, and each user can belong to many groups.
For instance, all system administrators and staff might belong to the staff group. They have access to private, group-only directories and files that all non-members (non-staff users) can’t access. In the same way, zoe’s manager and the office manager might be members of the zoe group – to have group-mode access to some or all of her files and directories.
Changing group ownership
If you belong to a group (and you always belong to at least one group: yours), you can change the group owner of any file or directory you own with the chgrp command.
For instance, zoe might keep confidential notes on system users in a file named users.xml. She’d use:
chgrp staff users.xml
to set the file’s group owner to staff. Then she’d use:
chmod 750 users.xml
(or chmod u=rw,g=r,o= users. xml) to give herself read and write access, give read access to members of the group staff, and no access to other users. Any staff member can open the file, but other users have no access.
But which group?
If you belong to several groups, which group owns files you create? By default, it’s your primary group – the group listed first in the id output, after gid=. That’s usually not what you want in a directory shared by all users from a particular group.
For example, the directory /projectdir/4staff might have the permissions drwxrwx— so all members of the staff group can create, remove, and rename files in that directory. But if zoe creates a file in that directory with her default group of zoe, other members of the staff group probably can’t read it (unless they’re also members of the zoe group).
Set Group ID
The answer is to set the directory’s setgid bit, like the example below. (This bit is shown by s in the permission string drwxrws—.)
Files (and subdirectories) created in a directory with its setgid bit on will have the same group owner as their directory. So, with the following settings, any file created in /projectdir/4staff will automatically have the group owner staff:
chmod g+s /projectdir/4staff/
ls -ld /projectdir/4staff/
drwxrws— 5 zoe staff /projectdir/4staff/
Self-protection
If you’re the only user of your Unix system, and your filesystems aren’t networked to (or from) other computers, you may wonder why you need to understand permissions. One good reason is to prevent accidents – unintended deletes or changes to your own files and directories. Mistakes happen, so protecting yourself makes sense.
Unwritable
If report.doc is an important file, you can make it unwritable by everyone – including yourself – with:
chmod a-w report.doc
But setting permissions on the file’s contents don’t control whether the file can be removed or replaced!
For instance, if you run gedit report.doc, gedit might rename report.doc to report.doc.bak, then create a new report.doc file. That’s perfectly legal.
To protect against all changes, the file’s directory also needs to be unwritable.
Attributes
The permissions scheme we’ve seen works on Unix systems, in general, as well as on Linux. Some Unix filesystems – the widely-used ext2 and ext3 types, for instance – also have other file attributes.
The manual pages for lsattr and chattr explain how to list and change file attributes.
Here’s a quick example. The superuser can use chattr to set the a attribute, which makes a file append-only: the file can’t be modified, only added to. Running lsattr shows this attribute, but ls -l doesn’t.
# chattr +a log.txt
# exit
$ ls -l log.txt
-rw-rw-r– 1 zoe staff 1168 Oct 30 12:29 log
$ lsattr log.txt
—–a– log.txt
$ cp /var/log/dmesg log.txt
cp: overwrite `log’? y
cp: cannot create regular file `log’: Operation not permitted
$ cat /var/log/dmesg >> log.txt
$ ls -l log.txt
-rw-rw-r– 1 zoe staff 3109 Oct 30 12:33 log.txt
Permissions
What are files?
As we’ve discussed before, files are globs of data, in either text or binary form.
File names can be up to 256 characters long with “-“, “_”, and “.” characters along with letters and numbers.
When you do a long file listing, you see 10 characters on the left that indicate file type and permissions:
drwxrwxrwx
What are permissions?
File and directory permissions define your access to those files or directories.
There are actually three sets of permissions:
User/owner/author’s
Group’s
Other users’
And there are three distinct permissions for each user or group:
Read
Write
Execute
which determine what level of access they have.
Default permissions
Every user’s profile defines what permissions will be given to files and folders created by that user.
When the user is created, a home directory for that user is created at /home/username, and the initial contents are copied from /etc/skel. (You can change these content files to change new user defaults.)
Triple Triplets: User, Group, Others
When you run an ls command, user permissions for directories look like:
drwxrwxrwx
User permissions for files look like:
-rwxrwxrwx
The first character in the field indicates a file type of one of the following:
d = directory
l = symbolic link
s = socket (a connection to send data from a process on one computer to a process on another)
p = named pipe (a connection to send data from one command or process to another; represented as a file)
– = regular file
c= character (unbuffered) device file
b=block (buffered) device file
Types of permissions
There are 5 possible characters in the permission fields. They are:
r = read – This is only found in the read field.
w = write – This is only found in the write field.
x = execute – This is only found in the execute field.
s = setgid/setuid – This is only found in the user or group execute field
t = set text attribute, which shows up in the others execute field
If there is a “-” in a particular location, there is no permission given. This may be found in any field whether read, write, or execute field.
Owner permissions
The first rwx refers to the owner’s permissions.
By default, the owner gets full permissions: read, write and execute:
rwx
Group permissions
The second triplet describes the permissions set for the default group of the owner, for instance:
r-x
which allows group members to read and execute the file, but not write to it.
(Generally a new group is created for every new user, but you can override this behavior and place your new user in another group.)
Others’ permissions
The last triplet describes the permissions granted to any other user. You might, for instance, only allow them to read your files, but deny them write or execute access:
r–
Permissions are additive
If a user is both a member of the Group-permission group, and is (always, of course) a member of Others, the permissions that user will have are additive, which is to say they add together, rather than subtracting from each other.
Note the permissions of this file:
-rwxr–r-x root wheel … myfile.txt
Root owns the file; the wheel group has read permissions; Others have execute permissions. A member of the wheel group will get both the read permission from the Group triplet, and the execute permission from the Others triplet.
Changing permissions
The shell command chmod changes file and directory permissions.
Generally you’ll use numeric permissions, though you can use alphabetic mode to modify permissions also (see man chmod.)
chmod 755 ~/myfile.sh
Numeric permissions
Numeric permissions are calculated by adding individual permissions.
4 means “read”
2 means “write”
1 means “execute” or “browse” (depending on whether you’re changing a file or a directory).
It all adds up
Add 1, 2 and 4 to describe a permission set.
1+4 = 5 (read and execute/browse)
1+2 = 3 (read and write)
1+2+4 = 7 (read, write, execute/browse)
Three triplets, three numbers
Remember, you can set groups for the owner, their group, and other users. This means you’ll use three numbers to describe a full set of permissions:
chmod 755 ~/myfile.txt
sets permissions to:
-rwxr-xr-x
Careful with that directory!
The “x” permission has special consequences for directories. You might think you should eliminate “execute” permissions for directories that don’t contain executables.
That would be wrong. If you deny “x” permissions on a directory, users will not be able to browse to that directory.
Directory Permissions
Directory permissions don’t work the same way file permissions do. In particular, the execute permission is very different.
Permission
Meaning for files
Meaning for directories
read
User can open and read a file
User can list contents of a directory – only if user also has execute permission!
write
User can open, read and edit file
User can add files to or remove files from the directory
execute
User can execute the file (if it’s executable)
User can enter directory and work with its contents. This is sometimes called the search permission.
Important Note: A user’s permissions on a directory do not transfer to the contents of the directory! Individual file permissions still prevail.
Some common permissions
chmod 777 ~/myfile.txt (= rwxrwxrwx)
chmod 755 ~/myfile.txt (= rwxr-xr-x)
chmod 644 ~/myfile.txt (= rw-r–r–)
chmod 000 ~/myfile.txt (= ———)
When should you allow execute permissions?
There are two times you’ll want to give users an “x” permission:
When you want to let them enter and browse directories.
When a file is a shared executable, for instance a Perl file used in a web site.
Alphabetic Permissions
So far, we’ve described permissions using numbers, which works in most cases.
Be aware that you can modify permissions by “adding” or “subtracting” individual permissions.
For instance, if you wanted to remove execute permissions (for yourself) from a file named myfile, you could issue the command:
chmod u–x myfile
Effectively, you just “subtracted” execute permission.
Notice that in that last command,
chmod u–x myfile
we didn’t specify a user name. That’s because we just modified the owner’s permissions.
If we want to modify the group’s permissions, the command becomes:
chmod g-x myfile
Even more, if a category of user is not specified, ALL users are assumed:
chmod -x myfile
removes everyone’s execute permission.
Why the heck do we want to do this?
You can use this feature for quick modifications of permissions, but its real use is in setting the
User Identification Attribute
(Let that one sink in.)
Let’s get abstract. Under Unix, every file (which means everything) is owned by someone.
Further, every process that’s executed is also owned by someone.
The trick question is, who “are” you as you run an executable file that belongs to someone else?
The answer is, it depends.
Under most circumstances, you execute executables as yourself, with all the permissions (and restrictions) you have.
However, there are some programs that have to be run as someone else. Consider services, for instance.
Daemons
What is a service? A service is a utilitarian program that is run by a daemon, which is the process that actually sits and waits for a signal. The daemon provides the service.
So what? Here’s what: daemons run even when no one is logged in. They’re running in kernel mode as opposed to user mode.
And what does “daemon” mean? Disk Access and Execution MONitor
Set the User ID Attribute
Recall that the file permissions bits include an execute permission bit for file owner, group and other.
When the execute bit for the owner is set to “s” the set-user-ID bit is set. This causes any persons or processes that run the file to have access to system resources as though they are the owner of the file. When the execute bit for the group is set to “s”, the set-group-ID bit is set and the user running the program is given access based on access permission for the group the file belongs to. The following command:
chmod u+s myfile
sets the user ID bit on the file “myfile.”
The ping command is an example of a command with SUID. ping can be called by anyone, but it runs as root – it has to, for some of its functionality.
NOTE: You can only SUID compiled binary programs on some platforms. Test this on yours.
Set the Group ID Attribute
The command:
chmod g+s /somedir
sets the group ID bit on the directory “/somedir”.
These two items have the group or user ID bit set:
-rws–x–x 1 root root 14024 Sep 9 1999 chfn
-rwxr-sr-x 1 root mail 12072 Aug 16 1999 /locks
Set UID/GID Examples
-rws–x–x 1 root fred 14024 Sep 9 1999 chfn
-rwxr-sr-x 1 root mail 12072 Aug 16 1999 /locks
The “s” takes the place of the normal location of the execute bit in the file listings above.
This special permission mode has no meaning unless the file has execute permission set for either the group or other as well. This means that in the case of the lockfile, if the other users (world execute) bit is not set with permission to execute, then the user ID bit set would be meaningless since only that same group could run the program anyhow.
In both files, everyone can execute the binary. The first program, when run is executed as though the program is the root user. The second program is run as though the group “mail” is the user’s group.
So Please Note: SUID/SGID will only work if execute permissions on lower triplets are set as well.
Careful with that axe
For system security reasons it is not a good idea to set many program’s set user or group ID bits any more than necessary, since this can allow an unauthorized user privileges in sensitive system areas. If the program has a flaw that allows the user to break out of the intended use of the program, then the system can be compromised.
Special Directory Permissions
There are two special bits in the permissions field of directories. They are:
s – Set group or user ID (SGID/SUID)
t – Save text attribute – The user may delete or modify only those files in the directory that they own or for which they have write permission.
Directory Set Group ID
If the setgid bit on a directory entry is set, files in that directory will have the group ownership of the directory, instead of the group of the user who created the file.
Special Directory Permissions
This attribute is priceless when people are working on projects.
If users work in a directory with the setgid attribute set, any files created in the directory by any of the users will have the permission of the directory group, rather than the user’s default (primary) group.
For example, the administrator creates a group called project and adds the users Kathy and Mark to the group project. The directory projectdir is created with the set GID bit set. Kathy and Mark, although in different primary groups, can work in the directory and have full access to all files in that directory – but still not be able to access files in each other’s primary group.
Setting the GID bit
Issue this command to set the GID bit on a directory:
chmod g+s /projectdir
Resulting in this directory listing of the directory projectdir:
drwxrwsr-x 2 kathy project 1674 Sep 17 1999 projectdir
The “s” in place of the execute bit in the group permissions causes all files written to the directory projectdir to belong to the group “project.”
Save text attribute (the “sticky bit”)
The /tmp directory is typically world-writable and looks like this in a listing:
drwxrwxrwt 13 kathy project 4096 Apr 15 08:05 tmp
Everyone can read, write, and access the directory. The “t” indicates that only the user who created a file in this directory can delete or modify that file. Root always can overrule this, of course.
Setting a sticky bit
To set the sticky bit in a directory, issue the following command:
chmod +t directory_name
So why would you do this?
This setup is one step “tighter” than the set GID setup.
If you didn’t create the file, you can’t change or delete the file. But otherwise you can work freely in the directory.
If you set the GID bit instead, anyone with access to the directory can modify anyone else’s files. However, in many programming situations, this is a good thing.
Using numerical values for special permissions
You can use the numerical values of chmod to set special permissions:
4 = SUID
2 = SGID
1 = Sticky Bit
So, for example, you could set the sticky bit using:
chmod 1755
simply by placing an additional number to the left of the permission set.
Examples of making changes to permissions
chmod u+x myfile
Gives the user (owner) execute permission on myfile.
chmod ugo+x myfile
Gives everyone execute permission on myfile.
chmod 400 myfile
Gives the user read permission, and removes all other permission.
chmod 764 myfile
Gives user full access, group read and write access, and other read access.
chmod 751 myfile
Gives user full access, group read and execute permission, and other, execute permission.
chmod u+s myfile
Set the setuid bit.
chmod go-rx myfile
Remove read and execute permissions for the group and other.
umask
Quick: when you create a file, who owns the file?
What group “owns” the file?
What are its default permissions?
We’re into the realm of a concept called the umask: the “user mask.”
The umask is a numerical value that describes the default permissions any file you create will have.
You can see your own current umask with the command:
umask
The numbers look similar to file permission numbers, like 421. However, they function differently.
About masks in general
A user mask is just one kind of mask you’ll find in the world of Unix.
Another common one is the subnet mask, a number like:
255.255.255.0
Here’s the thing to know:
Masks are subtractive, not additive.
The subnet mask 255.255.255.0 actually means (in the case of an IP address), “subtract this value from the maximum possible values.”
In this case it means, “mask the first three octets (triplets) of local IP addresses.” In other words, if my IP is 192.168.0.5, and I’m trying to reach 192.168.0.7, the address I’ll really look for is simply 0.0.0.7.
If you’re not familiar with this concept in IP addressing, don’t worry about it.
Instead, consider the highest possible level of permissions a file can have, expressed as numbers:
777 for executable files
666 for text files
Note that contemporary Linux distros won’t even allow you to create a file as executable by default. This is a good idea.
Better than 666
777, as a file permissions descriptor, means the user, group and others all have read, write and execure permissions.
I don’t want everyone to have full permissions to my files. I want them to have less. In fact, for most cases, I want them to have:
700
However, this may be too draconian, so I’ll settle for:
755 (for executables)
644 (for text files)
umask: the numerical value
So, if I want to “lower” the default permissions that a file I create will have, I have to subtract from 777 (for executables) or 666 (for text files.)
If I’m keeping full permissions for myself, the first number of my umask will be 0.
If I’m restricting my group members to reading and executing, but not writing to my files, the second number of my umask will be 2 (thus subtracting 2, or the “write” value) for any file I create.
And if I’m willing to give the same read/execute permissions to others, the third number of my umask will also be 2 (thus subtracting write permissions).
Setting my umask
You can set the value of your shell’s umask from any shell prompt.
umask is a built-in bash shell command. The preferred way to set it is to add a umask command to the .profile or .bashrc file in your home directory.
To change it system wide there is a file in the /etc directory hierarchy where the default values for user shells are set.
System default umask
The system default umask is set in the file /etc/bashrc, which is generally modifiable only by system admins.
Its current default value is probably the best one for system wide use. For specific groups of users you may want a slightly more open value, but it shouldn’t be needed most of the time.
Your own umask
You can override the system default umask by setting your own in your home directory.
Modify either ~/.bashrc or ~/.bash_profile (preferably the second) to change your own umask permanently.
Use the umask command to change it only for the current session:
umask 022
Think like chmod
The three digits of a umask are written in the same order as you’d specify with chmod: first, the user mask; then, the group; finally, all others.
So, a umask of 002 denies no (0) permission for the user owner and group owner, and denies write (2) permission for others. A umask of 027 would deny no (0) permission for user, write (2) permission for group, and all (7, or 4+2+1) permissions for other users.
umask by example
An easy way to see what a umask does is to set it, create an empty file and directory (with touch and mkdir), and then use ls -l to see the permissions (here we’re using the shell’s semicolon (;) operator to run two commands from a single prompt):
umask 027
touch myfile; mkdir ymdir
ls -l myfile; ls -ld mydir
-rw-r—– glenn glenn … myfile
drwxr-x— glenn glenn … mydir
rm myfile; rmdir mydir
Filesystems
The Filesystem Tree
The Unix and Linux filesystem is based on the Filesystem Heirarchy Standard (FHS). (You can research it at http://www.pathname.com/fhs/ .)
There are no C: drives, D: drives or any other lettered drives. Instead, all disks are mounted into a common filesystem tree. You can even mount other computers into this tree. Everyone shares the same filesystem (which is why permissions are so important).
See the Partitioning section of the Installation page for information about setting up partitions.
See the excellent Novell page, File System Primer, at http://wiki.novell.com/index.php/File_System_Primer#File_System_Comparison
Review the list on the page above, then note these filesystems not included in the Novell list:
| Additional Unix/Linux Filesystems | |
| Type | Description |
| bfs | Boot filesystem – used at startup on most Unix systems |
| cdfs | Compact Disk filesystem – one way to access CD data as normal files |
| iso9660 | “The” CD-ROM filesystem, based on an ISO standard |
| udf | Universal Disk Format filesystem – used during the creation and burning of CDs |
| minix | The original proto-Linux filesystem |
| msdos | FAT 16-bit filesystem (not FAT32) |
| vfat | FAT compatible filesystem with long filename support |
| VxFS | HP-UX filesystem |
For a more comprehensive list of filesystem types, see Guide to Linux Filesystem Mastery by Sheryl Calish at http://www.oracle.com/technology/pub/articles/calish_filesys.html
Learn about Journaling File Systems from Steve Best’s article at
http://www.linux-mag.com/content/view/1167
Journaling File Systems in particular benefit from performance tuning: See Steve Best’s article “Tuning Journaling File Systems” at
http://www.linux-mag.com/id/2927/
You’re likely to need to make a file system on (format) a floppy, if it’s new (is there still such a thing?). Use the mkfs command. For a “Linux-only” floppy, you could use ext2 as your file system:
mkfs -t ext2 /dev/fd0
If it needs to be usable under Windows, use FAT instead:
mkfs -t msdos /dev/fd0
If you don’t specify, you’ll get ext2:
mkfs /dev/fd0
Formatting a Partition (creating a filesystem)
Filesystem Creation Commands Command Purpose Usage mkfs Makes filesystems of most types mkfs -t ext2 mkfs.msdos
mkdosfs
mkfs.vfatCreate a DOS FAT filesystem mkdosfs /dev/fd0 mkfs.ext2
mke2fsCreates an ext2 filesystem mkfs.ext2 /dev/hda2 mkfs.ext3
mke2fs -jCreate an ext3 (journaled) filesystem mkfs.ext3 /dev/hda2 mkisofs Creates a CD-ROM filesystem mkisofs -o iso_file starting_directory mkfs.reiserfs
mkreiserfsCreata a REISERFS filesystem mkreiserfs /dev/hda3
ext2 and ext3
Use tune2fs to “adjust tunable filesystem parameters” on ext2 and ext3 filesystems. These include:
- The number of mounts between filesystem checks
- The maximum time between filesystem checks
- Set the volume label
- Reserve blocks for the filesystem or by group
- Add journaling, which turns an ext2 filesystem into an ext3 filesystem.
See man tune2fs for details before using this command.
fsck
fsck is the generic filesystem check utility. There are actually whole families of filesystem check utilities, specific to certain filesystems. fsck is actually a “pass-through” utility that invokes the filesystem-specific checkers, which have names like fsck.fstype.
See a typically excellent page at SS64.com:
http://ss64.com/bash/fsck.html
for a simple rundown. Of course see man fsck for full details (which will refer you through to filesystem-specific details).
fsck wants you to provide a filesystem type. If you don’t it assumes ext2.
fsck -t ext3 /dev/hda1
Filesystem-specific Check Commands Command Purpose Usage e2fsck Checks ext2 and ext3 filesystems e2fsck /dev/fd0 badblocks
e2fsck -cChecks ext2 and ext3 filesystems for bad blocks badblocks /dev/fd0
e2fsck -c /dev/fd0reiserfsck Checks REISERFS filesystems reiserfsck /dev/hdc1 reiserfsck -c Checks REISERFS filesystems for bad blocks reiserfsck -c /dev/hdc1 mkisofs Creates a CD-ROM filesystem mkisofs -o iso_file starting_directory mkfs.reiserfs
mkreiserfsCreata a REISERFS filesystem mkreiserfs /dev/hda3
fsck only performs a quick check unless you specify the -f “full” option:
fsck -f -t ext3 /dev/hda1
Common fsck Options Option Description -f Full check -a Automatically repair errors -A Check all filesystems marked with a 1 or 2 in /etc/fstab -Cf Do a full check and display progress line -AR Check all filesystems marked with a 1 or 2 in /etc/fstab, except the / filesystem -V Verbose output
Note that you can only check unmounted filesystems. Be particularly clear that you can’t fsck the root filesystem while it’s mounted (that is, while your system’s running). (See http://www.cs.uu.nl/wais/html/na-dir/unix-faq/unixware/qt-faq.html – search for fsck when you arrive.)
While the defaults can vary, an ext2 or ext3 filesystem will force a check every 20-40 mounts (which usually means reboots), or every 180 days, whichever comes first. This is one place you can use tune2fs:
tune2fs -i 90d /dev/fd0
The command above, for instance, changes the check interval to 90 days (90d). You can use seconds (the default), days (d), weeks (w) or months (m). A setting of 0 disables interval-based testing.
If fsck can’t repair a file, it places it in the lost+found directory, giving the file its inode number for its name.
Disk Usage
The df (disk free space) command is the first tool to know. It gives you a fast audit of free space. Use it by itself:
df
or with human-readable output:
df -h
in HP-UX:
ddf
Perhaps more useful is the du (directory usage) command. Specify a directory to analyse:
du /usr
Or just get a summary:
du -s /usr
Make it human-readable:
du -hs /usr
Check used and available inodes on ext2 and ext3 filesystems with dumpe2fs. To keep it readable, always use the -h option:
dumpe2fs -h /dev/hda1