1. Getting to Know the Shell

The shell is an interpreter between the system kernel and the user, a bit like a translator. A script is a file in a particular language that runs in a predefined order, such as Shell, Python, Ruby, or JavaScript.

How to approach writing a script
Clarify the task steps->organize the statements->complete the file structure
For example, add a user named tom and set the password to 123123
Clarify the task steps:

  1. Add the user
    useradd tom

  2. Set the password
    passwd 123123

Organize the statements:

1
2
useradd tom         
echo 123123|passwd -stdin tom

Complete the file structure: 

1
2
3
4
5
6
7
# !/bin/bash
# 20141128,by beepy
echo "Creating user tom"
useradd tom
echo "Setting password"
echo 123123| passwd -stdin tom
echo "User created and password set"

Run the file

  • chmod +x uad,.sh./uad.sh

  • sh uad.sh

  • source uad.sh

2. Combining Shell Commands

Pipes: command1 | command2

Find aa in the test file
cat test | grep aa

Count the files ending in .conf under /etc
find /etc -name "*.conf” -type f | wc -l

View the httpd processes:
ps -aux | grep httpd

Redirection

Type Operator Purpose
Input redirection < Read data from the specified file
Output redirection <, << Overwrite or append output to the specified file
Standard error output 2>, 2>> Overwrite or append error messages to the specified file
Combined output &>, &>> Overwrite or append both standard output and errors to the specified file

Check the Linux kernel version
uname -r
Write the result to version.txt
uname -r > version.txt

Since there's no version2.txt, this gives an error
cat version.txt version2.txt

How do you send the error message to a file? Use 2>
cat version.txt version2.txt 2> error.txt

Logical separators

  • AND: &&
  • OR: ||
  • Command separator: ;

Logical AND
echo "haha" && echo "heihei"

Logical OR
echo "haha" || echo "failed"

Use logical AND or OR to check whether a command succeeded. Both of these check whether a directory was created successfully
mkdir test/a 2>/dev/null && echo "success"
mkdir test/a 2>/dev/null || echo "failed"

The command separator separates multiple commands
cd boot/grub;ls -ls grub.conf

3. Shell Variables

Definition and assignment
Format: name=value

Referencing variables
Format: $name, ${name} 

1
2
name = wangbin		
echo $name

Double quotes: allow references and \ escapes

1
2
echo “$name"
wangbin

Double quotes reference directly and output the value of $name

Single quotes: no references, no escapes

1
2
echo '$name'
$name

Single quotes can't reference, so the output is still the literal $name

Backticks `` or $( ): followed by a command

1
2
var = `uname -r`
echo $var

The command above outputs the Linux kernel version directly

Environment variables
Used to record or set runtime parameters; env lists all environment variables

  • Set by the system: USER, LOGNAME, HOME, SHELL…
  • Set by the user: PATH, LANG, CLASSPATH…

echo $USER $HOME $SHELL

Controlled by the system or script; can't be assigned manually

$?: whether the previous command succeeded; returns a status code, 0 for success, non-zero for errors
$0: the script's own name
$1-$9: the first through ninth positional parameters
$*: all positional parameters on the command line
$#: the number of positional parameters on the command line

mkdir test
echo $?, success, returns 0
0
mkdir /1/2/3,
echo $?, since 1 and 2 don't exist, creation fails and returns non-zero
1

Write a script that uses the variables above:

1
2
3
4
5
#!/bin/bash
echo "Program name: $0"
echo "Number of positional parameters: $#"
echo "The first positional parameter is: $1"
echo "All positional parameters: $*"

4. Arithmetic and Number Handling

Integer arithmetic
Two formats:
the expr command evaluates an expression
expr number1 operator number2
outputs the number directly

The $[] expression, arithmetic substitution
$[number1 operator number2]
the result needs echo to output

Operators: + - * / %

expr 322 / 322: since is a wildcard in Linux, it must be escaped when used as a multiplication sign

A few number-handling tips

Increment and decrement variables
let var++
let var–

1
2
3
4
x = 10,y = 30
let x++;echo $x
let y—;echo $y
let x+=2;echo $x increments by 2

Generate random numbers with RANDOM, ranging from 0 to 32767
echo $RANDOM

Take the remainder of a random number to get a random number from 0 to 99
echo $[RANDOM%100]

seq generates number sequences
seq first last
seq first step last

seq 4
seq 3 6
seq 3 3 19

Decimals
The shell can't handle decimals directly; you need the bc command. For example, to calculate with decimals
echo 433.44-43.5 | bc
Calculate with decimals and keep four decimal places
echo "scale = 4;10/3" | bc

5. String Handling

Substrings

  1. Splitting paths: the dirname and basename commands
  2. expr: expr substr $var1 start length, numbering starts at 1
  3. The ${} expression: ${var:start:length}, numbering starts at 0

dirname for the path, basename for the file name 

1
2
3
var = “/etc/beep/test.txt"
dirname $var = “/etc/beep/"
basename $var = “test.txt"

Substring with expr

1
2
var = “hahaheiheihehehehehehe"
expr var 4 5

Substring with ${}

1
2
3
var = “hahaheiheihehehehehehe"
echo ${var: 4:6}
echo ${var ::5} ; if omitted, it starts from 0

String replacement

Replacement with the ${} expression
${var/old/new} replaces only the first old with new
${var//old/new} replaces every old in the variable; use two slashes //

Random strings

How random strings are generated:

  1. /dev/urandom (generates random characters of all kinds)
  2. /usr/bin/md5sum (turns them into standard encoded characters)
  3. /bin/cut (cuts out the characters you need)

Example:
First convert the random device characters to ASCII characters and assign them to var, then cut them
echo $var | cut -b start-end; both start and end can be omitted.

1
2
3
head -1 /dev/urandom | md5sum
var = head -1 /dev/urandom | md5sum
head -1 /dec/urandom | md5sum | cut -b -8

6. Conditional Tests

What a test is
A test is a command; its $? return value tells you whether the condition holds

Format
Two formats: test expression
or [ expression ], with spaces inside the brackets

How to practice
Follow it directly with && echo YES to see the result
[ expression ] && echo YES

File status checks
-e: whether the target exists
-d: whether it's a directory
-f: whether it's a file

[-f “/boot/fstable”] && echo yes
[-f “/etc”] && echo yes

Permission checks
-r: whether it's readable
-w: whether it's writable
-x: whether it's executable 

1
2
3
4
ls -l /etc/shadow # stores account and password info; only the root user can read it
[-x  "/etc/shadow"] && echo  YES # check for execute permission
[-r  "/etc/shadow"] && echo  YES # check for read permission
[-w  "/etc/shadow"] && echo  YES # prints YES; write permission is an exception for the owner

Integer comparison

-eq: equal
-ne: not equal
-gt: greater than
-lt: less than
-ge: greater than or equal
-le: less than or equal 

1
2
3
who| wc -l # count logged-in users; $() means an expression
[$(who | wc -l) -eq 2] && echo YES # check whether the number of users equals 2
[$(who | wc -l) -gt 2] && echo YES # check whether the number of users is greater than 2

String matching
= the two strings are equal
!= the two strings are not equal 

1
2
[$USER = “root] && echo YES
[$USER != “root”] && echo YES

7. if Statements

Single-branch if statement

1
2
3
if condition
     then commands
fi

Example: check the backup directory /opt/repo and create it if it doesn't exist.

1
2
3
4
5
6
7
8
9
10
11
BACKUP_DIR = “/opt/repo"
if [  ! -d $BACKUP_DIR ]
     then mkdir -p $BACKUP_DIR
fi
Two-branch if statement
if condition
          then commands
          else other commands
fi

Example: check whether a target host is alive and show the result

1
2
3
4
5
6
7
ping -c 3 -i 0 2 -W $1 &> /dev/null
if [$? -ew 0]
then
     echo “host $1 is up"
else
     echo “host $1 is down"
fi

Multi-branch if statement

if condition1
then commands1
elif condition2
then commands
…
else
commandsN
fi

Example: grade a score as excellent / pass / fail

1
2
3
4
5
6
7
8
read -p "Enter a score (0-100): " grade
if [ $grade -gt 85] && [ $grade -lt 100]
     then echo "$grade, excellent"
elif [ $grade -gt 60] && [ $grade -lt 75]
     then echo "$grade, pass"
else
     echo "$grade, fail"
fi

8. for Loops

Syntax

1
2
3
4
for var in list
do
     commands
done

Check whether IP addresses are in the host address list

1
2
3
4
for IP in host_list
do
     check status
done

Usage examples:

1
2
3
4
for i in “1st” “2nd” “3rd"
do
     echo $i
done

Output the contents of hosts word by word

1
2
3
4
for i in $ (cat /etc/hosts.conf)
do
     echo $i
done

Add user accounts in bulk from users.txt, one per line, setting the initial password to 123456 and requiring a change at first login.

1
2
3
4
5
6
for i in $(cat /root/users.txt)
do
     useradd $i
     echo “123456” | passwd —stdin $i
     chage -d 0 $i
done

Check the status of hosts in an IP range, 192.168.4.1 - 192.168.4.10
to see whether they respond to ping.

1
2
3
4
5
6
7
8
9
10
11
IP_PRE = “192.168.4."
for IP in $(seq 1 5)
do
    ping -c 3 -i 0 2 -W 3 $[IP_PRE]& IP &> /dev/null
     if [ $? -eq 0];then
          echo “$[IP_PRE]$IP is up"
     else
           echo “$[IP_PRE]$IP is down"
     fi
done

9. case Statements

Syntax

1
2
3
4
5
6
7
8
9
10
11
case value in
pattern1)
     commands
     ;;
pattern2)
     commands2
     ;;
…...
*)
     default commands
esac

A simple service control example

1
2
3
4
5
6
7
8
9
10
11
case control_arg in
start)
          start the service
          ;;
stop)
          stop the service
          ;;
*)
          show the script's usage
esac

Examples

Identify the type of user input

1
2
3
4
5
6
7
8
9
10
11
read -p "Enter a character and press Enter: " key
case “$key” in
     [a-z]|[A-Z])
          echo "You entered a letter"
     ;;
     [0-9])
          echo "You entered a digit"
     ;;
     *)
          echo "You entered some other control character"
esca

Write a service script, sleepd

* Responds to the start and stop control arguments
* Hands the service over to chkconfig for management

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
chkconfig: - 90 10
#description :Daemon script sleepd server
case “$1” in
start)
     sleep 3600 & , run in the background
     ;;
stop)
     pkill -x “sleep"
     ;;
*)
     echo "Usage: $0 [start | stop]
esca

chkconfig setup
Set the chkconfig parameters at the top of the script
Add it as a system service

Run this in the /etc/init.d/sleepd directory
chkconfig -add sleepd

10. awk, the Text Processing Tool

What it does
awk is named after the first letters of its three authors' names
Checks input based on pattern matching
prints the desired results to the screen

Format
awk 'pattern{action}' file1 file2

awk 'NR==1{print} /etc/hosts # prints the first line of hosts; NR is a built-in awk variable for the line number
You can also use head -1 /etc/hosts for the same result

A few common built-in variables:
NR the number of the line being processed (line number), number row
FS the field separator, space or tab by default
$n the nth field of the current line
$0 the entire text of the current line

Examples

1
2
3
4
5
test-awk.txt
1 This is a first line
2 Hello world
3 64.233.185.99   google.com
4 hunter:503:504:504::/home/sites:/bin/bash

Print lines 1 through 3
awk 'NR==1,NR==3'{print}' test-awk.txt

Print lines 1 and 3
awk 'NR==1||NR==3'{print}' test-awk.txt

Print all odd lines
awk 'NR%2==1{print}' test-awk.txt

Print all even lines
awk 'NR%2==0{print}' test-awk.txt

Regex matching uses two slashes: /chars/

List all lines containing a
awk '/a/{print}' test-awk.txt

List lines ending in bash
awk '/bash$/{print}' test-awk.txt

awk's default separator is space or tab, so you can print specific fields, for example fields 1 through 3 of lines 1 through 3

awk ‘NR==1,NR==3{print $1,$3} test-awk.txt

Besides the default separator, you can set one with -F. For example, using a dot as the separator, print the second field of the line whose third field is 185. $2 is the second field, and $0 prints the whole line
awk -F. '$3=="185”{print $2}' test-awk.txt

11. Text Processing with sed

sed stands for stream editor. It doesn't just print; it can also replace and delete, so it's more powerful. It filters, modifies, and deletes text based on pattern matching.

Format
sed 'command' file1 file2 …
sed -n 'command' file1 file2 ……, -n only prints the lines you ask for
sed -i 'command' file1 file2 ……, -i edits the file in place

Command format
[address1[,address2]]operation
Separate multiple commands with semicolons

The most common operations
p print lines
n read the next line (skip the current line)
d delete text
s substitute text
a append text

Examples

1
2
3
4
5
6
7
8
9
10
11
Print lines 3 through 5 of hosts
sed -n '3,5p' /etc/hosts
Print lines 3 and 5 of hosts
sed -n '3p;5p' /etc/hosts
Print every other line with sed: all odd lines
sed -n 'p;n' test-awk.txt
Print every other line with sed: all even lines
sed -n 'n;p' test-awk.txt

Note: separate n and p with a semicolon

Using regular expressions with sed
sed -n '/a/,$p' test-sed.txt
Prints from the first line containing a through the last line; $ means the last line

sed -n /\<This\>/p' test.txt
Prints lines containing the word This as a whole word; the backslashes escape the angle brackets

Delete matching lines

1
2
3
4
5
6
7
8
Delete lines 2-3
sed '2,3d' test.txt
Delete lines containing aa and the last line; again, $ means the last line
sed '/aa/d;$d' test.txt
Delete non-matching lines with !d
sed '/aa/!d' test.txt

Replace matching text

1
2
3
4
5
6
7
8
9
sed '3,4s/aa/bb/g' test.txt
s means substitute, and g means global (replace all)
Special replacement effects
sed '1,2s/^/#/g' test.txt
^ means the start of a line; replacing it with # comments out the line
sed 's/ter//g' test.txt
Delete the characters ter by replacing them with a space

None of the sed commands above use flags, so they only show the result without changing the file itself. To edit the file directly, use the -i flag.