Topics for Linux Beginners

 Fundamental command line skills

 Linux Networking Commands

 Linux System Administration

 Linux Users Administration

 Linux File Systems Administration

 Linux System Services

 Linux System Level Security

 Apache Web Server ( HTTP )

 Electronic Mail Servers ( SMTP )

 File Sharing Servers ( NFS )

 Samba Server ( SMB )

 File Transfer Protocol ( FTP )

 Domain Name Server ( DNS )

 Package Management


Site is under construction

Thank You

crontab command in Linux

Back

The /etc/crontab file is setup in a specific format. Each line can be blank, a comment a variable, or a command. Naturally Blank lines and comments are ignored. Users run  regular commands. Anyone who runs a regular command, weather it be you or a daemon, is limited by various environmental variables.

The format of a line in /etc/crontab is now detailed below. If you see any asterisk (*) in any coloumn, the daemon runs that aommand for all possible values of that coloumn, the cron daemon runs that comand for all possible values of that coloumn.

For example , an * in the minute field means that the command is run every minute during the specified hour (s).

Consider another example

1 5 3 4 * ls

This line runs the ls command every April 3 at 5.01 A.M. The (*) in the day of week column simply means that it does not matter what day of the week it is crontab still runs the ls command at the specified time.The entries associated with the cron daemon are flexible. 

For example

 A 7-10 entry in the hour field would run the specified command at 7:00 A.M  8:00 A.M  9:00 A.M  10:00 A.M . A list of entries in the minute field such as 0,5,10,15,20,25,30,35,40,45,50,55 would run the specified command every five minutes. But that's a lot of numbers. The */5 in the minute field would lead to the same result. The cron daemon also recognizes abbreviations for months and the day of the week.

Columns in a cron Configuration file

minute 
hour day of month  
month  
day of week 
command


1. Scheduling a Job For a Specific Time

The basic usage of cron is to execute a job in a specific time as shown below. This will execute the Full backup shell script (full-backup) on 10th June 08:30 AM.

Please note that the time field uses 24 hours format. So, for 8 AM use 8, and for 8 PM use 20.

30 08 10 06 * /home/shabbir/full-backup
30 – 30th Minute
08 – 08 AM
10 – 10th Day
06 – 6th Month (June)
* – Every day of the week

2. Schedule a Job For More Than One Instance (e.g. Twice a Day)

The following script take a incremental backup twice a day every day.

This example executes the specified incremental backup shell script (incremental-backup) at 11:00 and 16:00 on every day. The comma separated value in a field specifies that the command needs to be executed in all the mentioned time.

00 11,16 * * * /home/shabbir/bin/incremental-backup
00 – 0th Minute (Top of the hour)
11,16 – 11 AM and 4 PM
* – Every day
* – Every month
* – Every day of the week

3. Schedule a Job for Specific Range of Time (e.g. Only on Weekdays)

If you wanted a job to be scheduled for every hour with in a specific range of time then use the following.

Cron Job everyday during working hours

This example checks the status of the database everyday (including weekends) during the working hours 9 a.m – 6 p.m

00 09-18 * * * /home/shabbir/bin/check-db-status
00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
* – Every day of the week

Cron Job every weekday during working hours

This example checks the status of the database every weekday (i.e excluding Sat and Sun) during the working hours 9 a.m – 6 p.m.

00 09-18 * * 1-5 /home/shabbir/bin/check-db-status
00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
1-5 -Mon, Tue, Wed, Thu and Fri (Every Weekday)

4. How to View Crontab Entries?

View Current Logged-In User’s Crontab entries
To view your crontab entries type crontab -l from your unix account as shown below.

$ crontab -l

@yearly /home/shabbir/annual-maintenance
*/10 * * * * /home/shabbir/check-disk-space

[Note: This displays crontab of the current logged in user]
View Root Crontab entries
Login as root user (su – root) and do crontab -l as shown below.

root@dev-db# crontab -l
no crontab for root

Crontab HowTo: View Other Linux User’s Crontabs entries
To view crontab entries of other Linux users, login to root and use -u {username} -l as shown below.

root@dev-db# crontab -u sathiya -l
@monthly /home/sathiya/monthly-backup
00 09-18 * * * /home/sathiya/check-db-status

5. How to Edit Crontab Entries?

  Edit Current Logged-In User’s Crontab entries
To edit a crontab entries, use crontab -e as shown below. By default this will edit the current logged-in users crontab.

$ crontab -e

@yearly /home/shabbir/centos/bin/annual-maintenance
*/10 * * * * /home/shabbir/debian/bin/check-disk-space
~
"/tmp/crontab.XXXXyjWkHw" 2L, 83C

[Note: This will open the crontab file in Vim editor for editing.
Please note cron created a temporary /tmp/crontab.XX... ]
When you save the above temporary file with :wq, it will save the crontab and display the following message indicating the crontab is successfully modified.

~
"crontab.XXXXyjWkHw" 2L, 83C written
crontab: installing new crontab
Edit Root Crontab entries
Login as root user (su – root) and do crontab -e as shown below.

 root@dev-db# crontab -e

Edit Other Linux User’s Crontab File entries
To edit crontab entries of other Linux users, login to root and use -u {username} -e as shown below.

root@dev-db# crontab -u sathiya -e

@monthly /home/sathiya/fedora/bin/monthly-backup
00 09-18 * * * /home/sathiya/ubuntu/bin/check-db-status
~
~
~
"/tmp/crontab.XXXXyjWkHw" 2L, 83C

6. Schedule a Job for Every Minute Using Cron.

Ideally you may not have a requirement to schedule a job every minute. But understanding this example will will help you understand the other examples mentioned below in this article.

* * * * * CMD
The * means all the possible unit — i.e every minute of every hour through out the year. More than using this * directly, you will find it very useful in the following cases.

When you specify */5 in minute field means every 5 minutes.
When you specify 0-10/2 in minute field mean every 2 minutes in the first 10 minute.
Thus the above convention can be used for all the other 4 fields.

7. Schedule a Background Cron Job For Every 10 Minutes.

Use the following, if you want to check the disk space every 10 minutes.

*/10 * * * * /home/shabbir/check-disk-space

It executes the specified command check-disk-space every 10 minutes through out the year. But you may have a requirement of executing the command only during office hours or vice versa. The above examples shows how to do those things.

Instead of specifying values in the 5 fields, we can specify it using a single keyword as mentioned below.

There are special cases in which instead of the above 5 fields you can use @ followed by a keyword — such as reboot, midnight, yearly, hourly.

Table: Cron special keywords and its meaning
Keyword Equivalent
@yearly 0 0 1 1 *
@daily 0 0 * * *
@hourly 0 * * * *
@reboot Run at startup.

8. Schedule a Job For First Minute of Every Year using @yearly

If you want a job to be executed on the first minute of every year, then you can use the @yearly cron keyword as shown below.

This will execute the system annual maintenance using annual-maintenance shell script at 00:00 on Jan 1st for every year.

@yearly /home/shabbir/red-hat/bin/annual-maintenance

9. Schedule a Cron Job Beginning of Every Month using @monthly

It is as similar as the @yearly as above. But executes the command monthly once using @monthly cron keyword.

This will execute the shell script tape-backup at 00:00 on 1st of every month.

@monthly /home/shabbir/suse/bin/tape-backup

10. Schedule a Background Job Every Day using @daily

Using the @daily cron keyword, this will do a daily log file cleanup using cleanup-logs shell scriptat 00:00 on every day.

@daily /home/shabbir/arch-linux/bin/cleanup-logs "day started"

11. How to Execute a Linux Command After Every Reboot using @reboot?

Using the @reboot cron keyword, this will execute the specified command once after the machine got booted every time.

@reboot CMD

12. How to Disable/Redirect the Crontab Mail Output using MAIL keyword?

By default crontab sends the job output to the user who scheduled the job. If you want to redirect the output to a specific user, add or update the MAIL variable in the crontab as shown below.

ramesh@dev-db$ crontab -l
MAIL="ramesh"

@yearly /home/shabbir/annual-maintenance
*/10 * * * * /home/shabbir/check-disk-space

[Note: Crontab of the current logged in user with MAIL variable]

If you wanted the mail not to be sent to anywhere, i.e to stop the crontab output to be emailed, add or update the MAIL variable in the crontab as shown below.

MAIL=""

13. How to Execute a Linux Cron Jobs Every Second Using Crontab.

You cannot schedule a every-second cronjob. Because in cron the minimum unit you can specify is minute. In a typical scenario, there is no reason for most of us to run any job every second in the system.

14. Specify PATH Variable in the Crontab

All the above examples we specified absolute path of the Linux command or the shell-script that needs to be executed.

For example, instead of specifying /home/shabbir/tape-backup, if you want to just specify tape-backup, then add the path /home/shabbir to the PATH variable in the crontab as shown below.

$ crontab -l

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/home/shabbir

@yearly annual-maintenance
*/10 * * * * check-disk-space

[Note: Crontab of the current logged in user with PATH variable]

15. Installing Crontab From a Cron File

Instead of directly editing the crontab file, you can also add all the entries to a cron-file first. Once you have all thoese entries in the file, you can upload or install them to the cron as shown below.

$ crontab -l
no crontab for ramesh

$ cat cron-file.txt
@yearly /home/shabbir/annual-maintenance
*/10 * * * * /home/shabbir/check-disk-space

$ crontab cron-file.txt

$ crontab -l

@yearly /home/shabbir/annual-maintenance
*/10 * * * * /home/shabbir/check-disk-space
Note: This will install the cron-file.txt to your crontab, which will also remove your old cron entries. So, please be careful while uploading cron entries from a cron-file.txt.

Automate System Administration : Cron and at commands in Linux

Back 

The cron system is essentially a smart alarm clock. When the alarm sounds, Linux runs the commands of your choice automatically. You can set the alarm clock to run at all sorts of regular time intervals. Many cron jobs are scheduled to run during the middle of the night, when user activity is lower. Of course, that timing can be adjusted. Alternatively the at system allows user to run commands of their choice, Once, at specified time in the future.

The cron daemons starts jobs on a regular schedule. The anacron system helps the cron daemon work on systems that are powered off at night. This helps enterprises that want to save energy.

It's configured to check the /var/spool/cron directory for jobs by user. In addition, it in co-operates jobs defined in the /etc/anacrontab file, based on 0anacron script in the /etc/cron.hourly directory. It checks  for scheduled jobs for the computer described in the /etc/crontab file and in the /etc/cron.d directory.

Types of Automate System administration commands 

Crontab

Anacron 

star command in Linux

Back

The star command is more appropriate for archiving files in a Selinux system. As the star command is not normally installed, you'll need to install it, one method is tyhe following command

# yum install star

Unfortunately, the star command does not quite work in the same fashion as tar. If you ever have to use the start command, some practice is appropriate. 

For example :
The following command would create an archive , with all Selinux contexts , from the current /home directory.

# star -xattr -H=exustar -c -f=home.star /home/

The -xattr switch saves the extended attribute associated with ACL's. The - c creates a new archives file. The -f specifies the name of the archive file.

Once the archive is created, it can be unpacked with the following command which extracts the archive.

# star -x -f=home.star

The star - x command can detect and restore files from archives configured with various compression schemes.


Star by default uses a fifo to optimize data flow from/to tape. This results in a normally streaming tape during the whole backup. See -fifo and fs= option to get information on how to find the best fifo size.

Star has no limitation on file-name length. Path names and link names up to PATH_MAX (1023 bytes with old OS versions and 4095 bytes with POSIX.1-2001) may be archived. Later versions may be able to deal with longer path names.

Star makes it easy to repair corrupted file systems. After a fsck -y has been run on the file system, star is able to restore only the missing files automatically. Use then b to check for differences. 

star automatically recognizes the type of the archive. star therefore is able to handle features and properties of different archive types in their native mode, if it knows about the peculiarities of the archive type. 

See the H=headertype option for more details. To be able to do this, star adds hidden fingerprints to the archive header that allows to recognize all star specific archive formats. The GNU tar format is recognized by the way it deviates from the standard.

star automatically recognizes and handles byte swapped archives. There is no option to manually control byte swapping.

star automatically recognizes and handles compressed archives inside plain files.

tar command in Linux

Back

The tar command was originally developed for archiving data to tape drives. However, it's commonly used for collecting a series of files, especially from a directory. 

For example : 
The following command backs up the information from the /home directory in the home.tar.gz file

# tar czvf home.tar.gz /home

This particular command creates (c) an archive, compress (z) it, in verbose (v) mode, with the file-name (f) that follows. Alternatively, you can extract (x) from that file with the following command.

# tar xzvf  home.tar.gz  /home

The compression specified (z) is associated with the gzip command, if you wanted to use bzip2 compression, substitute the j switch. But there are drawbacks to the tar command, as such archives do not store access control list settings or selinux attributes. But if a tar archives is all that's available, you can use commands like restorecon, to restore the contexts of an archive that have been restored to their original directories.


The tar (i.e., tape archive) command is used to convert a group of files into an archive.

An archive is a single file that contains any number of individual files plus information to allow them to be restored to their original form by one or more extraction programs. Archives are convenient for storing files as well as for for transmitting data and distributing programs. Moreover, they are very easy to work with, often much more so than dealing with large numbers of individual files.

Although tar was originally designed for backups on magnetic tape, it can now be used to create archive files anywhere on a filesystem. Archives that have been created with tar are commonly referred to as tarballs.

Unlike some other archiving programs, and consistent with the Unix philosophy that each individual program should be designed to do only one thing but do it well, tar does not perform compression. However, it is very easy to compress archives created with tar by using specialized compression utilities.

tar's basic syntax is :

tar option(s) archive_name file_name(s)

tar has numerous options, many of which are not frequently used. Unlike many commands, tar requires the use of at least one option, and usually two or more are necessary.

tar files are created by using both the -c and -f options. The former instructs tar to create an archive and the latter indicates that the next argument (i.e., piece of input data in a command) will be the name of the new archive file. 
Thus, for example : the following would create an archive file called file.tar from the three files named file1, file2 and file3 that are located in the current directory (i.e., the directory in which the user is currently working):

# tar -cf file.tar file1 file2 file3

It it not absolutely necessary that the new file have the .tar extension; however, the use of this extension can be is very convenient because it allows the type of file to be visually identified. It is necessary, however, that the -f option be the final option in a sequence of contiguous, single-letter options; otherwise, the system will become confused as to the desired name for the new file and will use the next option in the sequence as the name.

The -v (i.e., verbose) option is commonly used together with the -c and -f options in order to display a list of the files that are included in the archive. In such case, the above example would become

# tar -cvf file.tar file1 file2 file3

tar can also be used to make archives from the contents of one or more directories. The result is recursive; that is, it includes all objects (e.g., directories and files) within each level of directories. 

For example, the contents of two directories named dir1 and dir2 could be archived into a file named dir.tar with the following:

# tar -cvf dir.tar dir1 dir2

It is often convenient to use tar with a wildcard (i.e., a character which can represent some specific class of characters or sequence of characters). 
The following example uses the star wildcard (i.e., an asterisk), which represents any character or sequence of characters, to create an archive of every object in the current directory:

# tar -cf *

By default, tar creates an archive of copies of the original files and/or directories, and the originals are retained. However, they can be removed when using tar by adding the --remove-files option.

As it has no compression and decompression capabilities of its own, tar is commonly used in combination with an external compression utility. A very handy feature of the GNU version (which is standard on Linux) is the availability of options that will cause standard compression programs to compress a new archive file as soon as it has been created. They are -j (for bzip2), -z (for gzip) and -Z (for compress). 

Thus, for example, the following would create an archive named files.tar.bz2 of the files file4, file5 and file6 that is compressed using bzip2:

# tar -cvjf files.tar.bz2 file4 file5 file6

tar can also be used for unpacking tar files. However, before doing this, there are several steps that should be taken. One is to confirm that sufficient space is available on the hard disk drive (HDD). Another is to move to an empty directory (which usually involves creating one with an appropriate name) to prevent the reconstituted files from cluttering up the current directory and overwriting any files or directories with same names that are in it. 

In addition, if the archive has been compressed, it must first be decompressed using the appropriate decompression program (which can usually be determined by the filename extension).

In order to unpack a tar file, the -x (for extract) and -f options are required. It is also common to add the -v option to provide a running listing of the files being unpacked. 

Thus, for example, to unpack the archive file.tar created in a previous example the following would be used:

# tar -xvf file.tar

Just as options are available to allow three compression programs to automatically compress newly created tar files, the same options can be used to have the compression programs automatically decompress tar files prior to extraction. Thus, for instance, the following would decompress and extract the contents of the compressed archive files.tar.bz2 that was created in an above example:

# tar -xjvf files.tar.bz2

Files can be added to an existing archive using the -r option. As is always the case with tar, it is also necessary to use the -f option to indicate that the following string (i.e., sequence of characters) is the name of the archive. For example, the following would append a file named file7 to file.tar:

# tar -rf file.tar file7

The --delete option allows specified files to be completely removed from a tar file (except when the tar file is on magnetic tape). However, this is different from an extraction, as copies of the removed files are not made and placed in the current directory. Thus, for example, the files file1 and file2 can be removed from file.tar with the following:

# tar -f file.tar --delete file1 file2

The -t option tells tar to list the contents of an uncompressed archive without performing an extraction. Thus, the following would list the contents of file.tar:

# tar -tf file.tar

One of the very few options that can be used alone with tar is --help, which provides a relatively compact listing of the numerous options that are available. Another is --version, which shows the version number for the installed tar program as well as its copyright information.

Archives and compression commands in Linux

Back

Linux includes a variety of comands to archive groups of files. Some archives can be processed into packages such as RPMs. Other archives are just used as backups.

 In either case, archives can be a terrific convenience, especially when compressd. To that end, this section explores those archives and compression commands specially cited in Linux. These "essential tools" includes

gzip

bzip2

tar

star

gzip command in Linux

Back

The gzip and bzip2 commands are functionally similar, as they compress and decompress files, using different algorithms. The gzip command uses the Lempel-Ziv algorithm, found in some Microsoft compression algorithms. The bzip2 command uses the Burrows-Wheeler block sorting algorithm . While they both work well, the bzip2 command makes a big file a bit smaller.

For example, either of the two following commands could be used to compress a big picture file named big.jpg.


# gzip big.jpg
# bzip2 big.jpg

It adds .gz or a .bz2 suffix to the file, compressed to the associated algorithms. With the -d switch, you can use the same commands to reverse the process.

# gzip -d big.jpg.gz
# bzip2 -d big.jpg.bz2


If the compressed file name is too long for its file system, gzip truncates it. gzip attempts to truncate only the parts of the file name longer than 3 characters. (A part is delimited by dots.) If the name consists of small parts only, the longest parts are truncated. 

For example, if file names are limited to 14 characters, gzip.msdos.exe is compressed to gzi.msd.exe.gz. Names are not truncated on systems which do not have a limit on file name length.

By default, gzip keeps the original file name and time-stamp in the compressed file. These are used when decompressing the file with the -N option. This is useful when the compressed file name was truncated or when the time stamp was not preserved after a file transfer.

Compressed files can be restored to their original form using gzip -d or gunzip or zcat. If the original name saved in the compressed file is not suitable for its file system, a new name is constructed from the original one to make it legal.

gunzip takes a list of files on its command line and replaces each file whose name ends with .gz, -gz, .z, -z, _z or .Z and which begins with the correct magic number with an uncompressed file without the original extension. gunzip also recognizes the special extensions .tgz and .taz as short-hands for .tar.gz and .tar.Z respectively. When compressing, gzip uses the .tgz extension if necessary instead of truncating a file with a .tar extension.

gunzip can currently decompress files created by gzip, zip, compress, compress -H or pack. The detection of the input format is automatic. When using the first two formats, gunzip checks a 32 bit CRC. For pack, gunzip checks the uncompressed length. The standard compress format was not designed to allow consistency checks. 

However gunzip is sometimes able to detect a bad .Z file. If you get an error when un-compressing a .Z file, do not assume that the .Z file is correct simply because the standard uncompress does not complain. This generally means that the standard uncompress does not check its input, and happily generates garbage output. The SCO compress -H format (lzh compression method) does not include a CRC but also allows some consistency checks.

Files created by zip can be uncompressed by gzip only if they have a single member compressed with the 'deflation' method. This feature is only intended to help conversion of tar.zip files to the tar.gz format. To extract zip files with several members, use unzip instead of gunzip.

bzip2 command in Linux

Back

The gzip and bzip2 commands are functionally similar, as they compress and decompress files, using different algorithms. The gzip command uses the Lempel-Ziv algorithm, found in some Microsoft compression algorithms. The bzip2 command uses the Burrows-Wheeler block sorting algorithm . While they both work well, the bzip2 command makes a big file a bit smaller.

For example, either of the two following commands could be used to compress a big picture file named big.jpg.


# gzip big.jpg
# bzip2 big.jpg

It adds .gz or a .bz2 suffix to the file, compressed to the associated algorithms. With the -d switch, you can use the same commands to reverse the process.

# gzip -d big.jpg.gz
# bzip2 -d big.jpg.bz2


bzip2 compresses files using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. The command-line options are deliberately very similar to those of GNU gzip, but they are not identical.

bzip2 expects a list of file names to accompany the command-line flags. Each file is replaced by a compressed version of itself, with the name "original_name.bz2". Each compressed file has the same modification date, permissions, and, when possible, ownership as the corresponding original, so that these properties can be correctly restored at decompression time.

bzip2 and bunzip2 will by default not overwrite existing files. If you want this to happen, specify the -f flag.

If no file names are specified, bzip2 compresses from standard input to standard output. In this case, bzip2 will decline to write compressed output to a terminal, as this would be entirely incomprehensible and therefore pointless.

bunzip2 (or bzip2 -d) decompresses all specified files. Files which were not created by bzip2 will be detected and ignored, and a warning issued. bzip2 attempts to guess the file-name for the decompressed file from that of the compressed file as follows:


      filename.bz2      becomes    filename 
      filename.bz        becomes    filename 
      filename.tbz2     becomes    filename.tar 
      filename.tbz       becomes    filename.tar 
      anyothername    becomes    anyothername.out

If the file does not end in one of the recognized endings, .bz2, .bz, .tbz2 or .tbz, bzip2 complains that it cannot guess the name of the original file, and uses the original name with .out appended.

As with compression, supplying no filenames causes decompression from standard input to standard output.

bunzip2 will correctly decompress a file which is the concatenation of two or more compressed files. The result is the concatenation of the corresponding uncompressed files. Integrity testing (-t) of concatenated compressed files is also supported.

You can also compress or decompress files to the standard output by giving the -c flag. Multiple files may be compressed and decompressed like this. The resulting outputs are fed sequentially to stdout. Compression of multiple files in this manner generates a stream containing multiple compressed file representations. Such a stream can be decompressed correctly only by bzip2 version 0.9.0 or later. Earlier versions of bzip2 will stop after decompressing the first file in the stream.

bzip2 will read arguments from the environment variables BZIP2 and BZIP, in that order, and will process them before any arguments read from the command line. This gives a convenient way to supply default arguments.

Compression is always performed, even if the compressed file is slightly larger than the original. Files of less than about 100 bytes tend to get larger, since the compression mechanism has a constant overhead in the region of 50 bytes.

As a self-check for your protection, bzip2 uses 32-bit CRCs to make sure that the decompressed version of a file is identical to the original. This guards against corruption of the compressed data, and against undetected bugs in bzip2 (hopefully very unlikely). 

The chances of data corruption going undetected is microscopic, about one chance in four billion for each file processed. Be aware, though, that the check occurs upon decompression, so it can only tell you that something is wrong. It can't help you recover the original uncompressed data. You can use bzip2 recover to try to recover data from damaged files.

Return values: 0 for a normal exit, 1 for environmental problems (file not found, invalid flags, I/O errors, &c), 2 to indicate a corrupt compressed file, 3 for an internal consistency error (eg, bug) which caused bzip2 to panic.

System Administration Commands in Linux

Back

There are several system administration commands associated with system resource management and archives. 

System resource management allows you to see what process are running, to check the resources they are running , and to kill or restart those processes. 

Archives commands support the consolidation of a group of files in single archive, which can then be compressed.

System resource management commands

Archives and compression commands

top Command in Linux

Back

The top command sorts active process first by their CPU load and RAM memory usage. It provides an overview of the current system status, starting with the current up time, number of connected users, active and sleeping tasks, CPU load, and more.

One problem with the top and ps command is that they display the status of process on a system as a snapshot in time. That may not be enough. Processes may load a system for just a blip of time, or even periodic blips in time.

 One way to find more information about the overall load on a system is with two commands from the sysstat package sar and iostat. That system activity information is logged courtesy of the sa1 and sa2 commands associated with the /etc/cron.d/systat script, which will be described shortly.



     

sar Command in Linux

Back

The sar command can be used to provide a system activity report. 

The sar command collect, report, or save UNIX / Linux system activity information. It will save selected counters in the operating system to the /var/log/sa/sadd file. From the collected data, you get lots of information about your server:

CPU utilization

Memory paging and its utilization

Network I/O, and transfer statistics

Process creation activity

All block devices activity

Interrupts/sec etc.

There are two ways to invoke sar.

sar followed by an option (without specifying a saXX data file). This will look for the current day’s saXX data file and report the performance data that was recorded until that point for the current day.

sar followed by an option, and additionally specifying a saXX data file using -f option. 

This will report the performance data for that particular day. i.e XX is the day of the month.

In all the examples below, we are going to explain how to view certain performance data for the current day. To look for a specific day, add “-f /var/log/sa/saXX” at the end of the sar command.

All the sar command will have the following as the 1st line in its output.

$ sar -u

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

Linux 2.6.18-194.el5PAE – Linux kernel version of the system.

(dev-db) – The hostname where the sar data was collected.

03/26/2011 – The date when the sar data was collected.

_i686_ – The system architecture

(8 CPU) – Number of CPUs available on this system. On multi core systems, this indicates the total number of cores.

1. CPU Usage of ALL CPUs (sar -u)

This gives the cumulative real-time CPU usage of all CPUs. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on the last field “%idle” to see the cpu load.

$ sar -u 1 3

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:27:32 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle

01:27:33 PM       all      0.00      0.00      0.00      0.00      0.00    100.00

01:27:34 PM       all      0.25      0.00      0.25      0.00      0.00     99.50

01:27:35 PM       all      0.75      0.00      0.25      0.00      0.00     99.00

Average:             all      0.33      0.00      0.17      0.00      0.00     99.50

Following are few variations:

sar -u  Displays CPU usage for the current day that was collected until that point.

sar -u 1 3  Displays real time CPU usage every 1 second for 3 times.

sar -u  ALL Same as “sar -u” but displays additional fields.

sar -u  ALL 1 3 Same as “sar -u 1 3″ but displays additional fields.

sar -u -f /var/log/sa/sa10  Displays CPU usage for the 10day of the month from the sa10 file.

2. CPU Usage of Individual CPU or Core (sar -P)

If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.

“-P ALL” indicates that it should displays statistics for ALL the individual Cores.

In the following example under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.

$ sar -P ALL 1 1

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:34:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle

01:34:13 PM       all     11.69      0.00      4.71      0.69      0.00     82.90

01:34:13 PM         0     35.00      0.00      6.00      0.00      0.00     59.00

01:34:13 PM         1     22.00      0.00      5.00      0.00      0.00     73.00

01:34:13 PM         2      3.00      0.00      1.00      0.00      0.00     96.00

01:34:13 PM         3      0.00      0.00      0.00      0.00      0.00    100.00

“-P 1″ indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).

$ sar -P 1 1 1

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:36:25 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle

01:36:26 PM         1      8.08      0.00      2.02      1.01      0.00     88.89

Following are few variations:

sar -P ALL  Displays CPU usage broken down by all cores for the current day.

sar -P ALL 1 3  Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).

sar -P 1  Displays CPU usage for core number 1 for the current day.

sar -P 1 1 3  Displays real time CPU usage for core number 1, every 1 second for 3 times.

sar -P ALL -f /var/log/sa/sa10  Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.

3. Memory Free and Used (sar -r)

This reports the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

$ sar -r 1 3

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:28:06 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact

07:28:07 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

07:28:08 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

07:28:09 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

Average:        6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

Following are few variations:

sar -r

sar -r 1 3

sar -r -f /var/log/sa/sa10

4. Swap Space Used (sar -S)

This reports the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

$ sar -S 1 3

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:31:06 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad

07:31:07 AM   8385920         0      0.00         0      0.00

07:31:08 AM   8385920         0      0.00         0      0.00

07:31:09 AM   8385920         0      0.00         0      0.00

Average:         8385920         0      0.00         0      0.00

Following are few variations:

sar -S

sar -S 1 3

sar -S -f /var/log/sa/sa10

Notes:

Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.

Use “sar -H” to identify the hugepages (in KB) that are used and available.

Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.

Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

5. Overall I/O Activities (sar -b)

This reports I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

tps – Transactions per second (this includes both read and write)

rtps – Read transactions per second

wtps – Write transactions per second

bread/s – Bytes read per second

bwrtn/s – Bytes written per second

$ sar -b 1 3

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:56:28 PM       tps      rtps      wtps   bread/s   bwrtn/s

01:56:29 PM    346.00    264.00     82.00   2208.00    768.00

01:56:30 PM    100.00     36.00     64.00    304.00    816.00

01:56:31 PM    282.83     32.32    250.51    258.59   2537.37

Average:          242.81    111.04    131.77    925.75   1369.90

Following are few variations:

sar -b

sar -b 1 3

sar -b -f /var/log/sa/sa10

Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

iostat Command in Linux

Back

In contrast to sar, the iostat command report more general input / output statistics for the system, not only for the CPU, but also for connected storage device, such as local drives and mounted shared NFS directories.


The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics.

 On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors. A device header row is displayed followed by a line of statistics for each device that is configured. When option -n is used, an NFS header row is displayed followed by a line of statistics for each network file-system that is mounted.

The interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the time since system start-up (boot). Each subsequent report contains statistics collected during the interval since the previous report. 

The count parameter can be specified in conjunction with the interval parameter. If the count parameter is specified, the value of count determines the number of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the iostat command generates reports continuously.


The iostat command generates three types of reports, the CPU Utilization report, the Device Utilization report and the Network Files-ystem report.

CPU Utilization Report :

The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. 

Device Utilization Report :

The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices for which statistics are to be displayed may be entered on the command line. Partitions may also be entered on the command line providing that option -x is not used. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. 

If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags use

Network Filesystem report :

The Network File-system (NFS) report provides statistics for each mounted network file-system. The report shows the following fields:

Filesystem:
This columns shows the host-name of the NFS server followed by a colon and by the directory name where the network file-system is mounted.

renice Command in Linux

Back

The nice and renice command can be used to manage the priority of different processes.While the nice command is used to start a process with a different priority, the renice command is used to change the priority of a currently running process. Process priority in Linux specify numbers that seem counter-intuitive. The range of available nice numbers can vary from  -20 to 19. A process given a  priority of -20 takes precedence over all other processes. In contrast, a process given a priority of 19 will wait until the system is almost completely free before taking any resources. The default nice number of a process is 0.

# nice - 19 ./intensivescript.

This command starts the noted script with the lowest possible priority. If started at night ,the script is run untill just about any other jobs, such as run in one of the /etc/cron. directories is scheduled for execution. As such scripts are run on a schedule, they normally should take priority over some user-configured programs.

Sometimes a program is just taking up too many resources. If you need to make sure that program continues to run, one step before killing the associated process is to lower its priority with the renice command. 

Normally, the easiest way to identify a process that's taking up too many resources is by its PID in the output to the top command. That PID number is in the left-hand column of he output. For example,if you identify a process that's monopolizing current CPU and memory resources, copy the PID number of that process. If that number were 1234, the following command would change the nice number of that process to -10, which gives that process a higher priority than the default.

# renice -10 1234


Use ps axl to display the nice value of all running process as shown below.

# ps axl

How to assign a low priority to a shell-script? (higher nice value)

In the example below, when I started the nice-test.sh script in the background, it took the nice value of 0.

# ./nice-test.sh &
 [3] 13009

# ps axl | grep nice-test
0   509 13009 12863  17   0  4652  972 wait   S    pts/1      0:00 /bin/bash ./nice-test.sh
[Note: 6th column with value 0 is the nice.]

Now, let us execute the same shell script with a different nice value as shown below.

# nice -10 ./nice-test.sh &
[1] 13016

# ps axl | grep nice-test
0   509 13016 12863  30  10  4236  968 wait   SN   pts/1      0:00 /bin/bash ./nice-test.sh
[Note: 6th column with value 10 is the nice value for the shell-script.]

How to assign a high priority to a shell-script? (Lower nice value)

In the following example, let us assign a nice value of -10 (minus 10) to the nice-test.sh shellscript.

# nice --10 ./nice-test.sh &
[1] 13021

$ nice: cannot set priority: Permission denied

Note: Only root user can set a negative nice value. Login as root and try the same. Please note that there is a double dash before the 10 in the nice command below.

# nice --10 ./nice-test.sh &
[1] 13060

# ps axl | grep nice-test
4     0 13060 13024  10 -10  5388  964 wait   S<   pts/1      0:00 /bin/bash ./n

nice Command in Linux

Back

The nice and renice command can be used to manage the priority of different processes.While the nice command is used to start a process with a different priority, the renice command is used to change the priority of a currently running process. Process priority in Linux specify numbers that seem counter-intuitive. The range of available nice numbers can vary from  -20 to 19. A process given a  priority of -20 takes precedence over all other processes. In contrast, a process given a priority of 19 will wait until the system is almost completely free before taking any resources. The default nice number of a process is 0.

# nice - 19 ./intensivescript.

This command starts the noted script with the lowest possible priority. If started at night ,the script is run untill just about any other jobs, such as run in one of the /etc/cron. directories is scheduled for execution. As such scripts are run on a schedule, they normally should take priority over some user-configured programs.

Sometimes a program is just taking up too many resources. If you need to make sure that program continues to run, one step before killing the associated process is to lower its priority with the renice command. 

Normally, the easiest way to identify a process that's taking up too many resources is by its PID in the output to the top command. That PID number is in the left-hand column of he output. For example,if you identify a process that's monopolizing current CPU and memory resources, copy the PID number of that process. If that number were 1234, the following command would change the nice number of that process to -10, which gives that process a higher priority than the default.

# renice -10 1234


Use ps axl to display the nice value of all running process as shown below.

# ps axl

How to assign a low priority to a shell-script? (higher nice value)

In the example below, when I started the nice-test.sh script in the background, it took the nice value of 0.

# ./nice-test.sh &
 [3] 13009

# ps axl | grep nice-test
0   509 13009 12863  17   0  4652  972 wait   S    pts/1      0:00 /bin/bash ./nice-test.sh
[Note: 6th column with value 0 is the nice.]

Now, let us execute the same shell script with a different nice value as shown below.

# nice -10 ./nice-test.sh &
[1] 13016

# ps axl | grep nice-test
0   509 13016 12863  30  10  4236  968 wait   SN   pts/1      0:00 /bin/bash ./nice-test.sh
[Note: 6th column with value 10 is the nice value for the shell-script.]

How to assign a high priority to a shell-script? (Lower nice value)

In the following example, let us assign a nice value of -10 (minus 10) to the nice-test.sh shellscript.

# nice --10 ./nice-test.sh &
[1] 13021

$ nice: cannot set priority: Permission denied

Note: Only root user can set a negative nice value. Login as root and try the same. Please note that there is a double dash before the 10 in the nice command below.

# nice --10 ./nice-test.sh &
[1] 13060

# ps axl | grep nice-test
4     0 13060 13024  10 -10  5388  964 wait   S<   pts/1      0:00 /bin/bash ./n

kill Command in Linux

Back

Sometimes, it's not enough to re-prioritize a process. Some processes can just overwhelm a system. With some other operating systems, such situations requires a reboot. Linux is different. In most cases, you can stop such difficult processes with the kill and killall commands. In many cases, you can kill process directly from the top task browser.

If there's a situation where a process is taking up a lot of memory or CPU, you could apply the kill command directly to a PID number. For example, the following command is equivalent to steps just described in the top task browser :

# kill 2971

The kill command can be run by the owner of a process from his account. Thus, user michael could run the kill 2971 command from his regular account, as he has administrative privileges over processes associated with his user name.

 The kill command can send a wide variety of signals to different process. For a full list, run the kill -l command. Before the advent of scripts in the /etc/init.d directory, the kill -l command was used to send a restart signal to service daemons. For example if the PID number of the process associated with the vsFTP server is 2059, the following command is functionally equivalent to the /etc/init.d/vsftpd restart command.

# kill -1 2059

With out the -l  switch ,the kill command , under normal circumstances, would stop the given process. In this case, it would stop the vsFTP server. But sometimes, processes get stuck in loops. In some such cases, the kill command does not work by itself. The process continues running . In that case you can try two things.

First you could try the kill -9 command, which attempts to stop a process "uncleanly". If it is successful, other related processes running under the same name. For example the Apache Web Server starts several processes that run simultaneously. It's at best inefficient to kill just one process; the following command would kill all currently running server processes, assuming no other issues.

# killall httpd