System Monitoring : du and df

du

The disk usage (du) utility tells you how much space on disk is being taken by files and directories. Try:

du
du -h
du -s
du -sh

See the man page for du and note the expressions you can use to exclude certain items.

This gives you, in various forms, the space used. But it doesn’t give you an idea of how much space you have remaining. For that, use the df command.

df

The disk free space utility (df) provides a nice chart of free and used space. Try:

df
df -h
df -l
df -i

(See http://en.wikipedia.org/wiki/Inode about inodes.)

 

Getting Graphical: Baobab

Go to http://www.marzocca.net/linux/baobab.html and read about this utility.

Use yum or go to a repository to get and install Baobab.

Open it with the command baobab .

Note sorting options.

You can scan the whole filesystem, a local directory or a remote directory. For remote scanning, Baobab supports SSH, FTP, WebDAV and Samba protocols.

System Monitoring : top

All you need to do to run top is command:

top

Top is an interactive tool. This means that you can continue to issue commands or options for top while it is running. Run top, then try:

  • u
  • d
  • m

Top is useful for finding resource-hogging processes so you can kill them, among other things.

There’s an exhaustive listing of command and display options at About.com: http://linux.about.com/od/commands/l/blcmdl1_top.htm.

The top command leads the list of “20 Linux System Monitoring Tools Every SysAdmin Should Know” by Vivek Gite, at http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

System Monitoring : ps

Review Chapter 3, Linux Files and Processes, in Linux System Administration

ps

The ps command provides a list of current processes.

  • A process is not a thread
  • Processes run until finished
  • Processes fork: parent and child processes
  • PID
  • PPID

Note how the output of ps lets you trace from parent to child.

The output of the basic ps command:

PID TTY TIME CMD
2712 pts/1 00:00:00 bash
2727 pts/1 00:00:00 ps

  • Process ID
  • Terminal
  • CPU time
  • Command

Try:

ps a #BSD-style options

ps u

ps x

ps aux

ps -f #SysV-style options

ps -e

ps -l

The output of ps aux:

USER

PID

%CPU

%MEM

VSZ

RSS

TTY

STAT

START

TIME

COMMAND

root

1

0.3

0.2

1744

568

?

S

08:19

0:02

init[5]

root

2

0.0

0.0

0

0

?

SN

08:19

0:00

[ksoftirqd/0]

root

3

0.0

0.0

0

0

?

S

08:19

0:00

[watchdog/0]

root

4

0.0

0.0

0

0

?

S<

08:19

0:00

[events/0]

root

5

0.0

0.0

0

0

?

S<

08:19

0:00

[khelper]

root

6

0.0

0.0

0

0

?

S<

08:19

0:00

[kthread]

root

8

0.0

0.0

0

0

?

S<

08:19

0:00

[kacpid]

root

63

0.0

0.0

0

0

?

S<

08:19

0:00

[kblockd/0]

root

112

0.0

0.0

0

0

?

S

08:19

0:00

[pdflush]

root

113

0.0

0.0

0

0

?

S

08:19

0:00

[pdflush]

  • User name
  • Process ID
  • % of CPU utilization
  • % of RAM utilization
  • Virtual memory size
  • Resident set size, the non-swapped physical memory that a task has used (in kiloBytes)
  • Terminal (TTY)
  • Status:
    • D – uninterruptible sleep
    • R – runnable (in queue)
    • S – sleeping
    • T – traced or stopped
    • Z – “zombie”
    • W – as no resident pages
    • < – high priority
    • N – low priority
    • L – has pages locked in memory
  • Start time
  • Total time
  • Command name

 

Killing processes

Kill a process by process ID:

kill 1188

Kill a process by name:

pkill httpd

Kill a process that won’t die:

kill -9 1188

Kill multiple processes with the same name:

killall bash

 

Finding processes for a user:

ps aux | grep root –

 

About the init process

Note that the init program/process controls all startup, maintenance and shutdown processes for the system. You’ll see a number in brackets after the init process, e.g. [5]. This is the runlevel at which init is currently running.

 

About the ps program

ps has been around a long time. There are a LOT of versions, and many, many option conventions. See the man page for discussion of the differences between System V and BSD options.

Managing Shared Libraries

In Windows we’re familiar with .dll files, Dynamically Linked Libraries. In Unix, these are generally known as Shared Objects, or .so files. For the most part, these contain functions used in common by many applications.

Some applications are statically linked. This means that all the libraries used by an app are compiled right into the app, which results in larger files.

Wherever possible, applications are dynamically linked, which means any necessary libraries are loaded as needed from the main system. Thus the computer system must have the necessary libraries, but only one copy for every app that uses them. This frees up disk space, among other things.

You can find out what libraries an executable requires with the ldd command:

ldd /usr/bin/X11/xterm

Programs that use libraries use the ld.so component, which is responsible for finding and loading the library. ld.so looks at the file /etc/ld.so.conf for a list of directories containing shared libraries, in a manner similar to the way a shell uses your PATH to search for command binaries. ld.so also looks in /usr/lib and /lib, regardless of what ld.so.conf says.

Use the ldconfig command to check all these libraries and directories, update any links as necessary, and cache libraries where necessary:

ldconfig

Logs & Logging

Log Files

Where are your log files?

By strict Unix/Posix standards, log files should be in /var/log/. Sometimes, as with some Apache installations, logs are in the installation directory instead. But generally you can go to /var/log/and find a whole plethora of system logs.

Some daemons have their own directories: /var/log/httpd/, /var/log/samba/, etc.

Note the consecutively numbered log files in several places. These are the result of log file rotation, which is necessary because files have a limit on their size. In Red Hat distros, the logrotate package sets up cron jobs to perform this rotation. The /etc/logrotate.conf configuration file and the configuration files in the /etc/logrotate.d/ directory set up options; by default, logs rotate every week for four weeks. (See Red Hat’s System Administration Guide.)

 

Which Log Files?

Which logs should you be watching? What should you look for in your logs? These are not simple questions.

How you answer the first question depends on what your server is doing. If it is serving web pages, providing DNS service, running FTP or transferring email, you already know which logs to watch.

The second question might be tougher. Often you’re combing your log files after a hack attempt or a successful break-in, so you know you’re looking for specific evidence, which will, once again, depend on the kind of server you’re running.

Common Linux Log Files in /var/log/
boot.log System startup messages
cron cron and at daemon messages
dmesg System startup hardware detection messages
maillog Sendmail daemon messages
secure Network access messages from sshd, xinetd, and others
wtmp History of all login sessions
rpmpkgs List of packages installed with RPM
xferlog FTP messages
Xorg.0.log
or
XFree86
X Windows messages
lastlog Use the lastlog command to view this stored info about users and their last login time.
messages Daemon startup messages

 

The System Log Daemon

This daemon controls almost all logging on your system. It is configured via the /etc/syslog.conf file, which uses the format:

facility.priority /var/log/<logfile>

The available priorities, in ascending order of urgency, are:

debug

info

notice

warning or warn

error or err

crit

alert

emerg or panic

 

Analysis Tools

Consider tools like Webalyzer (here, for source or here, for FC4 RPM). Webalizer provides charting and graphing capabilities, so you can see which pages on your web site (web server) are the most popular, for instance, in full-color pie charts, bar graphs and you-name-it.

Compare Logrep and its Logreplight version.

Also see AWStats, and its awesome live demo.

 

Monitoring and Alert Tools

A busy web server (or group of them, such as you’d find at an ISP) requires more than after-the-fact analysis.

LogDog is ” A daemon for monitoring syslogd messages and alerting administrators.” The key word here is “alerting” – this is a critical monitoring tool for systems monitored full-time. One Kevin Cox wrote up his experience setting up LogDog with Snort for real-time alerts here.

And then, of course, there’s Snort itself.

Check out Red Hat’s own Log Viewer. Unfortunately this tool is only available with Red Hat Enterprise Linux.

Finally, take a look at TripWire, a dual-license project. The Open Source version is here, and the corporate site here.

 

Log Rotation

The logrotate utility backs up and clears log files, based on configuration information stored in /etc/logrotate.confcat this file for examples.

On many Linux systems you will find that logrotate is run via a cron job stored as /etc/cron.daily/logrotate.

You can run it manually:

logrotate /etc/logrotate.conf

System V

Review Chapter 7, Linux Files and Processes, section: Terminating/Restarting Processes Using Scripts, and Chapter 3, Startup and Shutdown, “Initialization and Startup Scripts”, in Linux System Administration

Also see Mayank Sarup’s article at FreeOS.com

Runlevels

0 – Shutdown. Get here by running the command init 0.

1 – Single-user mode. init 1

2 – Multi-user, no networking mode. init 2

3 – Multi-user, with networking, command-line mode. init 3

4 – (Usually) unused mode. init 4

5 – Multi-user, with networking, X11 (GUI) mode. init 5

6 – Reboot mode. init 6

You can check your current runlevel, and the immediately preceding runlevel, with the runlevel command.

 

The inittab file

This file is /etc/inittab . cat or less this file, and note the Default Runlevel section.

See the line:

id:[3-5]:initdefault

Do NOT set initdefault to 0 or 6!

 

Startup and Shutdown Scripts

In a terminal window, go to /etc/rc.d .
Run an ls command, then cat each of these:

rc – controls all movement between runlevels

rc.sysinit – runs once on system startup

rc.local – runs once, after everything else, on system startup

 

Where the Scripts Live

Change directories to /etc/init.d and run an ls command.

These are the actual scripts. Cat a few and notice how they handle multiple conditions (similarly to a class): start, stop, reload and status.

 

Where the Scripts Are Called

Change directories to /etc/rc3.d and run an ls command.

“K” scripts kill processes (services); “S” scripts start processes. The following number is a priority number.

 

The ln Command

This is a good time to learn about the ln command.

 

Startup Logging

You can see the startup logging information (the “kernel ring buffer”) by running the command:

dmesg #or /bin/dmesg

You’ll see the information that scrolled past on the screen as your system booted.

 

To see information about the success or failure of startup scripts (the ones in /etc/rc.d/) look at the file /var/log/boot.log.

To see kernel and system messages, see the file /var/log/messages.

All system logs are rotated by the logrotate script, configured by the file /etc/logrotate.conf.

Finally, current information about the system is in the /proc (virtual) filesystem.

 

See these files in /proc/:

interrupts – IRQ info

cpuinfo – CPU info

dma – DMA info

ioports – I/O info

meminfo – Memory info: available, free, swap, cached

loadavg – System load avg

uptime – Time since boot

version – Kernel version

scsi – SCSI info, if any

ide – IDE info

net – Network info

sys – Kernel config parameters

 

Runlevel and Shutdown Commands
init [0-6]

Change the computer’s state to the designated runlevel

shutdown -h now Shut down, hard, now
shutdown -r now Shut down and reboot now
shutdown -h +5 Shut down in 5 minutes
shutdown -r+5 “The system is going down in 5 minutes” Shut down in 5 minutes after displaying a message to users
shutdown -c Cancel a shutdown in progress
halt Halt the system immediately
reboot Reboot the system immediately

Working as Root

Review Chapter 1, The Basics of System Administration, the section titled “Working on the System as Root,” pp. 14 ff., in Linux System Administration

The Implications of Working As Root

It’s fun to work as root on a Unix system. You have access to everything, including every command, directory, and function, as well as every users documents.

It’s also a disaster to work as root, as you’ll discover the first time you nuke a production server with a mistyped command. For example, if you issue this command as a regular user:

dmesg > /etc/passwd

You’ll get slapped on the wrist. If you issue it as root, it’s time to start rebuilding your system. (Don’t try this, even as an experiment.)

Among the things you should consider as a system administrator are, Who can su to root? Who can use sudo? From what terminals can root log in? And can root log in remotely?

Remote Root

In every Linux and Unix system I’ve ever seen, the default settings deny root remote access. This is exactly as it should be. If you change this setting, you have been warned. That’s it.

Further, if you try to log in remotely to a system as root, your action will be logged, your IP address will be logged, and someone scary is likely to show up at your door. (Don’t ask me how I know this.)

Which Terminals

Even if your system boots to a GUI (runlevel 5), you are always logged in to a terminal. By default, you have 12 tty terminals available. This doesn’t mean only 12 users can log in; in fact it means every user has many terminals available. Use

CTRL > ALT > F[1-12]

to switch terminals, ranging from tty0 through tty6 on some systems, or tty11 on others. (There are also multiple vc terminals available.)

The file /etc/securetty contains the list of terminals from which root can log in. By default, root can log in to any of these – but not remotely.

If you want to live dangerously, add an entry for pts/[0-11] (or ttyp[0-11] on older systems), and root can log in remotely. Don’t do this.

Linux Servers

Linux III : Supporting Linux Servers

UNM Continuing Education Course

Instructor: Glenn Norman

Text:

Objectives

An advanced understanding of Linux server support

Familiarity with support, maintenance and deployment of Linux servers

Completing preparation for the CompTIA Linux+ Certification Exam