Monday, July 15, 2019

AWK - Tutorial

###AWK - Intro####
Features:
 1. Reporter
 2. Field Processsor
 3. Supports Scripting
 4. Programming Constructs - such as loop,contidition
 5. Default delimiter is whitespace
 6. Suuports: Pipes, files, and STDIN as sources of input
 7. Automatically tokenizee processed collums/fields into the valiables: $1 $2...$n

Usage:
awk '{instructions}' file(s)
awk '/pattern/ {procedure} ' file
awk -f script_file file(s)

Tasks:
Note: $0 repreents the current record or row
1. Print entire row, one at a time, from an input file(animals.txt)
 a. awk '{ print $0 }' animals.txt

2. Print specifi colums from (animals.txt)
 a. awk '{print $1}' animals.txt - this prints the 1st column from the file

3. Print multiple columns from (animals.txt)
 a. awk '{print $1; print $2}' animals.txt
 b. awk '{print $1,$2}' animals.txt

4. Print columns from line containing 'deer'using RegEx supports
  b. awk '/deer/ {print $0}' animals.txt

5. Print columns from line containing digits
  b. awk '/[0-9]/ {print $0}' animals.txt

6. Remove blank lines with sed and pipe output to awk for proceeding
 a. sed -e /^$/d animals.txt | awk '/^[0-9]*$/ {print $0}' animals.txt

7. Print ALL lines beginning with the animal 'dog' case-insensitive
 a. awk '/^dog/I { print }' animals.txt


###Delimiters###
Default delimiter: whitespace (space, tabs)
Use: '-F' to influence the default delimiter

Task:
1. Parse /etc/passwd using awk
 a. awk -F: ' { print } ' /etc/passwd
 b. awk -F: ' { print $1, $5 } ' /etc/passwd

2. Support for charecter classes in  setting the default delimiter
 a. awk -F"[:;,\t]"


###Awk scripts####
Features:
 1. Ablity to organize patterns and procedures into a script file
 2. The passwd/procedures are much neater and easier to read
 3. Less information is placed on the command-line
 4. By default, loops through lines of input from various sources: STDIN, Pipe, Files
 5. # is the default comment character
 6. Able to perform mathces base on specific fields

Awk Scripts consists of 3 Parts:
 1. Before (denotes using: BEGIN) - Executed prior to F line of input being read
 2. During (Main Awk Loop) - Foucuses on looping through lines of input
 3. After (denoted using: END) - Executed after the LAST line of input has been processed
Note: BEGIN and END components of Awk scripts are OPTIONAL

Tasks:
1. Print to the screen some of useful information without reading input (STDIN, Pipe, or File)
 a. awk 'BEGIN {print " Testing Awk without input file"}'

2. Set system variables: FS to color in BEGIN block
 a. awk ' BEGIN { FS = ":" }'
 b. awk ' BEGIN { FS = ":" ;  print FS }'
 c. awk ' BEGIN { FS = ":" ; print "Testing";  print FS }'

3. Write script to extract rows which rows which contain 'deer' from animals.txt using RegEx
 a. awk -f animals.awk animals.txt

[root@localhost]# cat awk_scripts.awk
#This script parses document for items containging deer
BEGIN { print "Begin processing of various records" }
/deer/ { print }
END { print "process complted" }

awk 'BEGIN {FS = ":"; print "Staring Awk" } /^root/ { print} END {print "Ending"}'  /etc/passwd
awk 'BEGIN {FS = ":"; print "Staring Awk" }  { print $1 "--" $7 } END {print "Ending"}'  /etc/passwd --##will give user and their home dir from /etc/passwd

4. Parse /etc/passwd
 a. print entire lines { print }
 b. print specific columns - { print $1, $2}
 c. print specific columns for a specific user - /linuxcbt/ { print $1, $5}
 d. print specific columns for a specific user matching a given column - $1 ~ /root/ { print $1, $5}
 e. test column #7 for the sting 'bash' - $7 ~ /bash/ { print }

#To find bash enabled user
# awk 'BEGIN {FS = ":"; print "Staring Awk" } $7 ~ /bash/  {  print  } END {print "Ending"}'  /etc/passwd
Staring Awk
root:x:0:0:root:/root:/bin/bash
student:x:1000:1000:student:/home/student:/bin/bash
Ending

###Awk Variables###
Features  3 Types of valiables:
 1. System - i,e. FILENAME, RS,ORS...
 2. Scalars - i,e. a = 3
 3. Arrays - i,e. variable_name[n]

Note: Variables do not need to be declared. Awk, automatically registers them in memory
Note: Variable name ARE case-sensitive

System Variables:
 1. FILENAME - name of current input file
 2. FNR - used when multiple input files are used
 3. FS - field separator - defaults to whitespace - can be a single character, including via a RegEx
 4. OFS - default field separator - whitespace
 5. NF - number of fields in the current records
 6. NR - current record number (it is auto-summed when referenced in END section)
 7. RS - record separator - defaults to a newline
 8. ORS - output record seprator - defualts to a newline
 9. ARGV - array of command-line arguuments - indexed at 0, beginning with $1
 10. ARGC - total # of command-line arguments
 11. ENVIRON - array of environment variables for the current user

awk 'BEGIN {FS = ":"; print "Staring Awk" } $7 ~ /bash/  {  print; print FILENAME; print NF } END {print "Ending"}'  /etc/passwd  Staring Awk
Staring Awk
root:x:0:0:root:/root:/bin/bash
/etc/passwd
7
student:x:1000:1000:student:/home/student:/bin/bash
/etc/passwd
7
Ending

Task:
 1. print key system variables
  a. print FILENAME ( print anywhere after the BEGIN block)

Test.sed
######
BEGIN {FS = ":"; print "Staring Awk" }
$7 ~ /bash/  {  print }
END {print "Ending" ; print "Filename: " FILENAME}'}
#####

# awk 'BEGIN {FS = ":"; print "Staring Awk" } $7 ~ /bash/  {  print } END {print "Ending" ; print "Filename: " FILENAME}'  /etc/passwd
Staring Awk
root:x:0:0:root:/root:/bin/bash
student:x:1000:1000:student:/home/student:/bin/bash
Ending
Filename: /etc/passwd

 b. print NF - number of fields per record
 c. print NR - current record number

# awk 'BEGIN {FS = ":"; print "Staring Awk" } $7 ~ /bash/  { print NR  $1 } END {print "Ending" ; print "Filename: "
 FILENAME}'  /etc/passwd
Staring Awk
1root
20student
Ending
Filename: /etc/passwd


 d. print ARGC - returns total number of command-line arguments

Scalar Variables:
variable_name = value

age = 39

Note: Set scalars in the BEGIN section, however, they can be, if required, set in the main loop

[ ++age ] - increments variable 'age' by 1, for each iteration of the mail loop (components 2 of 3)

Set variables to string using double quotes:
fulname ="cenots linux"

Concatenate variables by sepration the values using a space

Example:-
# awk 'BEGIN {FS = ":"; print "Staring Awk" ; fullname = "centos" "redhat"} $7 ~ /bash/  { print NR  $1 } END {print "Ending" ;print "Name_variable: " fullname ; print "Filename: " FILENAME}'  /etc/passwd
Staring Awk
1root
20student
Ending
Name_variable: centosredhat
Filename: /etc/passwd

Array Variables:
 Feature:
  1. List of information
Task:
 1. Defint an anrray variable to store various ages
  a. age[0] = 50
 2. Use split funtion to auto-build an array
  a. arr1num = split(string, array, separator)

###Operators###
 Features:
  1. Provides comparison tools for expressions
  2. Generally 2 types:
   a. Relational  - ==, 1+, <, >, <=, >=, ~ (RegEx Matches), !~ (RegEx Does NOT Match)
   b. Boolean - ||(or), &&(AND), !(NOT) - combines comparisons

 3. Print something if the current recored number is > 10
  a.  NR > 10 { print "current Record Number is greater than 10" }
 4. Extract records with ONLY 2 fields
  a. NF == 2 { print }
 5. Find records that have at least 2 fields and are positioned at record 5 or higher
  a. NF >= 2 && NR >= 5

###Loops####
Features:
 1. Support for: while, do, and for

While:
 { while (NR > 10) print "Greater than 10" }

For:
  { for (i=1; i<=10; i++) print i }

Do: Performs the action carried-out by while at least once:
 do action while (condition)


####Processing Records with AWk###
Task:
 1. Process multiple delimiters in the same file (across records)
  a. awk -F "[;: ]" '{ print }" animals2.txt
  b. awk 'BEGIN { FS="[ :;]" }; { print $2 } ' file
  c. awk -f script_name file
 2. Process multiple delimiters on the same line
  a. Note: Script does NOT change, however, input files DOES
 3. Normalize the output Field Separator (OFS)
  BEGIN { OFS=":" }

 4. Build animalclasses array from the list of clases in animals2.txt
    a. { userclass[NR] = $2 }  \for (i=0; i <= NR; i++) \ print "User Info" i ":  " userclass[i] - Place in mail loop - builds userclass array
    a1. awk ' BEGIN { FS = "[ ;:]"; print "Begin Processing of various records"} { userclass[NR] = $1 } END { print "Process Complete" ;for (i=0; i <= NR; i++) print "User Info",i,":  " ,userclass[i] }'  /etc/passwd

 5. Extract Daemon tntries from /var/log/messages
   a. extract kernel messages
    a1. awk ' /kernel/ { print }' /var/log/messages
    a2. awk ' /kernel/ { print }' /var/log/messages | awk '$8 ~ /cpu/I { print }'
    a3. awk ' /kernel/ { print }' /var/log/messages | awk ' BEGIN {print "Process stated" } $8 ~ /cpu/ { print } END {print "Process completed"}'


####Printf Formatting####
 Feature:
  1. Ability to control the width of fields in the output

Usage:
 printf("format", arguments)
 Supported printf formats include:
  1. %c - ASCII characters
  2. %d - Decimals - NOT floating point values OR values to the right of the deicmal point
  3. %f - Floating point
  4. %s - Strings
Note: printf does NO print newline character(s)
This means you will need to indicate a newline charecter sequences: -n - in the "format" section of the printf

Note: Default output is right-justified. Use '-' to indicate left-justification
General format section:
[-]width.precision[cdfs]
width - influences the actual width of the colum to be output
procision - influences the number of places to the right of the decimal point
procision - ALSO influences the numer of stirngs to be printed from a string

Examples | Tasks:
 1. print "Testing printf" from the command-line
  a. awk ' BEGIN { printf("Testing printf\n") }'

 2. read 'file' and format the output with prinf
  a. awk ' BEGIN { printf ("Here is the output\n") } { printf("%s", $1)} '
  b. awk ' BEGIN  { FS = ":"  ; printf ("Here is the output\n") } { printf("%s  :  %s\n", $1,$5)} ' /etc/passwd

 3. Apply width and precision to task #2
  a. awk ' BEGIN  { FS = ":"  ; printf ("Here is the output\n") } { printf("%.4s  :  %s\n", $1,$5)} ' /etc/passwd
  b. awk ' BEGIN  { FS = ":"  ; printf ("Here is the output\n") } { printf("%20s\t %20s\n", $1,$5)} ' /etc/passwd

 4. Left-justify task #3
   a. awk ' BEGIN  { FS = ":"  ; printf ("Here is the output\n") } { printf("%-20s\t %20s\n", $1,$5)} ' /etc/passwd

 5. Parse animals_with_prices.txt
   a. awk ' BEGIN  { FS = ":"  ; printf ("Here is the output\n") } { printf("%-20s\t %d\n", $1,$2)} ' /etc/passwd

Realtime Examples:- 
    a. powermt display dev=$i| awk -F=  ' $1 ~ /Logical device ID/ {print $2 }'`
    b. cat emc_psedoname| awk '{print $1}'| awk -F/ '{printf ("%.10s\n", $3)}'

 6. Format using printf 'files'
   a. awk ' BEGIN { FS = "[ ;:]"; print "Begin Processing of various records"} { userclass[NR] = $1 } END { print "Process Complete" ;for (i=0; i <= NR; i++) printf("%-9s %-2d %-2s %-10s\n", "User Info",i,":  " ,userclass[i]) }'  /etc/passwd
   a1. awk ' BEGIN { FS = "[ ;:]"; print "Begin Processing of various records"} { userclass[NR] = $1 } END { print "Process Complete" ;for (i=0; i <= NR; i++) printf("%-9s %-2d %-2s %-8s\n", "User Info",i,":  " ,userclass[i]); print "Total users " NR }'  /etc/passwd


 7. Apply upper and lower-case formatting to printf values
  a. awk ' BEGIN { FS = "[ ;:]"; print "Begin Processing of various records"} { userclass[NR] = $1 } END { print "Process Complete" ;for (i=0; i <= NR; i++) printf("%-9s %-2d %-2s %-8s\n", "User Info",i,":  " ,toupper(userclass[i])); print "Total users " NR }'  /etc/passwd
  b. awk ' BEGIN { FS = "[ ;:]"; print "Begin Processing of various records"} { userclass[NR] = $1 } END { print "Process Complete" ;for (i=0; i <= NR; i++) printf("%-9s %-2d %-2s %-8s\n", "User Info",i,":  " ,tolower(userclass[i])); print "Total users " NR }'  /etc/passwd

 8. Format output from /var/log/messages
  a. Extract date, time, server and daemon cloumns, include a header
   awk ' BEGIN { printf ("%-16s %-15s %-10s\n", "Date", "Server", "Daemon" )} /kernel/ { printf("%3s %2s %8s %-15s %10s\n", $1,$2,$3,$4,$5)}' /var/log/messages


   

Create rpm and deb using fpm

Create rpm and deb using fpm  fpm -s dir -t rpm -n unbound-exporter -v 1.0 --prefix /usr/bin unbound_exporter   fpm -s dir -t rpm -n unbound...