of Fundamentals UNIX ECE 257
Week 4 Advanced file processing along with awk, sed and Perl
College of Engineering: Electrical and Computer Engineering Department
Introduction
File processing
Regular expressions cmp and diff Awk / gawk Sed / Grep /egrep /fgrep Perl
College of Engineering: Electrical and Computer Engineering Department
Regular expressions
Used to specify a set of items to use in file processing commands
Alteration Any Char Begin string End string Escape seq Grouping Optional Repeat Repeat 1 or more Set A|B|C use char A, B or C /b..t/ boot, bait, ... ^b line begins b b$ line ending in b \* * (ab)+ ab, abab, ababab,... ab? a, ab ab* a, ab, abb, abbb,.... ab+ ab, abb, abbb,... /[Hh]ello/ Hello, hello awk and egrep all all all ed, sed, vi all awk, egrep all all all
College of Engineering: Electrical and Computer Engineering Department
Regular Expressions
Meta characters . [] [^] * ^ $ \ \\ / matches any single character matches any character in brackets matches any character NOT in brackets matches zero or more occurrences preceding matches beginning of a line matches the end of a line cancels the special meaning of meta char following represents the backslash marks the beginning and end of a regular expression
College of Engineering: Electrical and Computer Engineering Department
Simple strings a, 123_aAcdE, NCAA, NHL, kangaroo Delimiters /a/, /123_aAcdE/, /NCAA/, /NHL/, /kangaroo/ Period .at matches Cat, Bat, aat, 1at, ... Character class [a-z] any one of the lower case characters Compliment [^a-z] any one character not a lower case character Repetition
[a-zA-Z][a-zA-Z]* match any letter followed by zero or more letters
Regular expressions
Anchors
^F matches the character F occurring at the beginning of a line ^... matches any three character string at the beginning of a line
College of Engineering: Electrical and Computer Engineering Department
Compressing files
To compress files use compress[options][file-list]
> compress -c main.c HM9t SDl7uih&M9 TP @AfL3oPDbG)!&$0:TG/c > compress -f main.c > ls chicago.c indiana.c indy.c main.c.Z makefile chicago.o indiana.o indy.o main.o trip > > compress -v main.c main.c: Compression: 16.78% -- replaced with main.c.Z > > uncompress main.c.Z > ls chicago.c indiana.c indy.c main.c makefile chicago.o indiana.o indy.o main.o trip >
College of Engineering: Electrical and Computer Engineering Department
Compressing multiple files
> compress -v chicago.c indiana.c indy.c main.c chicago.c: Compression: 7.09% -- replaced with chicago.c.Z indiana.c: Compression: 17.87% -- replaced with indiana.c.Z indy.c: Compression: 14.61% -- replaced with indy.c.Z main.c: Compression: 16.78% -- replaced with main.c.Z > uncompress -v chicago.c.Z indiana.c.Z indy.c.Z main.c.Z chicago.c.Z: -- replaced with chicago.c indiana.c.Z: -- replaced with indiana.c indy.c.Z: -- replaced with indy.c main.c.Z: -- replaced with main.c >
College of Engineering: Electrical and Computer Engineering Department
Concatenate compressed files
> zcat chicago.c.Z indiana.c.Z indy.c.Z main.c.Z
/* The chicago function */ void chicago(void) { printf("\nI'm waiting at O'Hare International,\n"); printf("once the busiest airport in the world.\n"); } /* The indiana function */ void indianapolis(void); void indiana(void) { printf("\nBack home again, Indiana.\n"); indianapolis(); printf("Wander Indiana--come back soon.\n"); } /* The indianapolis() function */ #define POP2000 1.6 /* population in millions year 2000 */ void indianapolis(void) { printf("\nWelcome to Indianapolis, Indiana.\n"); printf("Population: %f million. \n", POP2000); } /* Illustrate multiple source files */ void chicago(void); void indiana(void); int main(void) { chicago(); indiana(); chicago(); return 0; }
Zcat uses options -h -r -t display help info operate recursively on subdirectories test integrity of compressed files
College of Engineering: Electrical and Computer Engineering Department
Sorting UNIX Files
Sort lines in an ascii file
Sort[options][file-list]
Options -b -d -f +n1[-n2] -r ignore leading blanks sort according to usual alphabetical order lower and uppercase are equal specify a field as sort key starting with +n1 and finish with n2 sort in reverse order
College of Engineering: Electrical and Computer Engineering Department
Example use
> cat names.txt jerry Seinfield george Kastanza liu Hong ideas Jolly drew Carey lewis Stein oswald Hanson > > sort names.txt drew Carey george Kastanza ideas Jolly jerry Seinfield lewis Stein liu Hong oswald Hanson > > sort -r names.txt oswald Hanson liu Hong lewis Stein jerry Seinfield ideas Jolly george Kastanza drew Carey > > sort +1 names.txt drew Carey oswald Hanson liu Hong ideas Jolly george Kastanza jerry Seinfield lewis Stein >
College of Engineering: Electrical and Computer Engineering Department
awk or gawk file processing
The awk text-processing utility and language is useful for such tasks as: - Tallying information from text files and creating reports from the results.
- Adding additional functions to text editors like "vi". - Translating files from one format to another. - Creating small data repositories. - Performing mathematical operations on files of numeric data.
College of Engineering: Electrical and Computer Engineering Department
Data for example
metal weight oz date minted country of origin The file has the contents: gold gold silver gold gold gold gold silver gold silver silver gold gold 1 1 10 1 1 0.5 0.1 1 0.25 0.5 1 0.25 1 1986 1908 1981 1984 1979 1981 1986 1986 1986 1986 1987 1987 1988 USA Austria-Hungary USA Switzerland RSA RSA PRC USA USA USA USA USA Canada description American Eagle Franz Josef 100 Korona Ingot ingot Krugerrand Krugerrand Panda Liberty dollar Liberty 5-dollar piece Liberty 50-cent piece Constitution dollar Constitution 5-dollar piece Maple Leaf
College of Engineering: Electrical and Computer Engineering Department
I could then invoke Awk/gawk to list all the gold pieces as follows: >awk '/gold/' coins.txt Or >gawk '/gold/' coins.txt This tells Awk to search through the file for lines of text that contain the string "gold", and print them out. The result is:
gold gold gold gold gold gold gold gold gold 1 1 1 1 0.5 0.1 0.25 0.25 1 1986 1908 1984 1979 1981 1986 1986 1987 1988 USA Austria-Hungary Switzerland RSA RSA PRC USA USA Canada American Eagle Franz Josef 100 Korona ingot Krugerrand Krugerrand Panda Liberty 5-dollar piece Constitution 5-dollar piece Maple Leaf
College of Engineering: Electrical and Computer Engineering Department
suppose I only want to print the description field, and leave all the other text out. I could then change my invocation of gawk to: >gawk '/gold/ {print $5,$6,$7,$8}' coins.txt This yields: American Eagle Franz Josef 100 Korona ingot Krugerrand Krugerrand Panda Liberty 5-dollar piece Constitution 5-dollar piece Maple Leaf
College of Engineering: Electrical and Computer Engineering Department
This example demonstrates the simplest general form of an gawk program: gawk <search pattern> {<program actions>} gawk searches through the input file for each line that contains the search pattern. For each of these lines found, gawk then performs the specified actions. In the prior example, the action was specified as: {print $5,$6,$7,$8} The "$5", "$6", "$7", and "$8" are "fields", or "field variables"
College of Engineering: Electrical and Computer Engineering Department
Since "coins.txt" has the the structure: metal weight in ounces date minted country of origin description -- then the field variables are matched to each line of text in the file as follows: metal: weight: date: country: description: $1 $2 $3 $4 $5 through $8
College of Engineering: Electrical and Computer Engineering Department
New Example
Now suppose I want to list all the coins that were minted before 1980. I invoke gawk as follows: >gawk '{if ($3 < 1980) print $3, " This yields: 1908 1979 Franz Josef 100 Korona Krugerrand ",$5,$6,$7,$8}' coins.txt
College of Engineering: Electrical and Computer Engineering Department
print out how many coins are in the collection:
>g
awk 'END {print NR,"coins"}' coins.txt
Number of lines in the file
This yields: 13 coins
The first new item in this example is the END statement. To explain this, I have to extend the general form of an gawk program to: awk 'BEGIN {<initializations>} <search pattern 1> {<program actions>} <search pattern 2> {<program actions>} END {<final actions>}'
g
College of Engineering: Electrical and Computer Engineering Department
Suppose the current price of gold is $425 I want to figure out the approximate total value of the gold pieces in the coin collection. I invoke Awk as follows: >gawk '/gold/ {ounces += $2} END {print "value = $" 425*ounces}' coins.txt This yields: value = $2592.5
College of Engineering: Electrical and Computer Engineering Department
sed text processing
Sed works as follows:
it reads from the standard input, one line at a time. for each line, it executes a series of editing commands then the line is written to STDOUT
College of Engineering: Electrical and Computer Engineering Department
An Example
use the s command. s means "substitute" or search and replace. The format is 's/regular-expression/replacement text/{flags}'
>cat file.txt I have three dogs and two cats >sed -e 's/dog/cat/g' -e 's/cat/elephant/g' file.txt I have three cats and two elephants >
College of Engineering: Electrical and Computer Engineering Department
Substitute command
The format for the substitute command is as follows: [address1[ ,address2]] 's/pattern/replacement/[flags] ' file Flags:
n g p w file replace nth instance of pattern with replacement replace all instances of pattern with replacement write pattern space to STDOUT if a successful substitution takes place Write the pattern space to file if a successful substitution takes place
Note: Address1 and address 2 are the line numbers to start and end substitution
College of Engineering: Electrical and Computer Engineering Department
Delete command
The delete command is very simple in it's syntax: it goes like this [address1[ , address2 ] ]d
College of Engineering: Electrical and Computer Engineering Department
Example
>cat file http://www.foo.com/mypage.html >sed -e 's@http://www.foo.com@http://www.bar.net@' file http://www.bar.net/mypage.html Uses different delimiter @ can also use : ; or % as delimiters
-e is a command line switch, allows for the use of different delimiters
College of Engineering: Electrical and Computer Engineering Department
Example
>cat file the black cat was chased by the brown dog >sed -e 's/black/white/g' file the white cat was chased by the brown dog
-n is another command line switch, allows suppression of auto-line print
College of Engineering: Electrical and Computer Engineering Department
Example
>cat file the black cat was chased by the brown dog. the black cat was not chased by the brown dog >sed -e '/not/s/black/white/g' file the black cat was chased by the brown dog. the white cat was not chased by the brown dog.
Note: In this instance, the substitution is only applied to lines matching the regular expression not. Hence it is not applied to the first line.
College of Engineering: Electrical and Computer Engineering Department
>cat file line 1 (one) line 2 (two) line 3 (three) Example a >sed -e '1,2d' file line 3 (three) Example b >sed -e '3d' file line 1 (one) line 2 (two) Example c >sed -e '1,2s/line/LINE/' file LINE 1 (one) LINE 2 (two) line 3 (three) Example d >sed -e '/^line.*one/s/line/LINE/' -e '/line/d' file LINE 1 (one)
College of Engineering: Electrical and Computer Engineering Department
>cat file hello this text is wiped out Wiped out hello (also wiped out) WiPed out TOO! goodbye (1) This text is not deleted (2) neither is this ... ( goodbye ) (3) neither is this hello but this is and so is this and unless we find another g**dbye every line to the end of the file gets deleted >sed -e '/hello/,/goodbye/d' file (1) This text is not deleted (2) neither is this ... ( goodbye ) (3) neither is this
College of Engineering: Electrical and Computer Engineering Department
Grep / fgrep / egrep
How to find Stuff inside files
College of Engineering: Electrical and Computer Engineering Department
Searching inside files
To find items in files use:
grep[options] pattern [file-list] egrep[options][string][file-list] Fgrep[options][expression][file-list] Common Options
-c -i -l -n -s -v -w print the number pf matches lines only ignore the case letters during matching process print file names with matching lines print line numbers along with matched lines useful for shell scripts (suppress error messages) print non-matching lines search for the given pattern as a string
College of Engineering: Electrical and Computer Engineering Department
Examples
> cat names.txt jerry seinfield george kastanza liu hong ideas jolly drew Carey lewis stein oswald hanson jolly mon jim carey jim belushi > grep jim names.txt jim carey jim belushi > grep jolly names.txt ideas jolly jolly mon > > grep -n carey names.txt 9:jim carey > grep -in carey names.txt 5:drew Carey 9:jim carey > > grep -n include *.c files.c:1:#include <stdio.h> files.c:2:#include <strings.h> getuid.c:1:#include <stdio.h> getuid.c:2:#include <pwd.h> getuid.c:3:#include <grp.h> mathexample.c:1:#include <math.h> random.c:1:#include <stdlib.h> sort.c:1:#include <stdio.h> sort.c:2:#include <strings.h> >
College of Engineering: Electrical and Computer Engineering Department
Another Example
> grep '\<jo' names.txt ideas jolly jolly mon > > grep -v '\<jo' names.txt jerry seinfield george kastanza liu hong drew Carey lewis stein oswald hanson jim carey jim belushi > > grep "^j" names.txt jerry seinfield jolly mon jim carey jim belushi
College of Engineering: Electrical and Computer Engineering Department
File Cutting and Pasting
Purpose is to Cut or Paste fields in files Formats Cut: cut blist [-n] [file-list] cut clist [file-list] cut flist [-dchar] [-s] [file-list] Options:
-b list -c list -d char -f list -n -s treat each byte as a column and cut bytes in list treat each character as a column and cut char in list use the character `char' instead of TAB as separator cut fields specified in list do not split characters (use with b) do output not lines not having the delimiter character
College of Engineering: Electrical and Computer Engineering Department
Example
> cat names.txt jerry seinfield 123.111.2222 george kastanza 123.222.3333 liu hong 123.333.4444 ideas jolly 234.555.6666 drew Carey 234.666.7777 lewis stein 345.777.8899 oswald hanson 345.888.9900 jolly mon 456.789.0123 jim carey 555.666.7777 jim belushi 789.123.4444
> cut -f1,2 names.txt jerry seinfield george kastanza liu hong ideas jolly drew Carey lewis stein oswald hanson jolly mon jim carey jim belushi
College of Engineering: Electrical and Computer Engineering Department
Example use
> cut -d: -f5,1,6 /etc/passwd root:Super-User:/ daemon::/ bin::/usr/bin sys::/ adm:Admin:/var/adm lp:Line Printer Admin:/usr/spool/lp uucp:uucp Admin:/usr/lib/uucp nuucp:uucp Admin:/var/spool/uucppublic listen:Network Admin:/usr/net/nls nobody:Nobody:/ noaccess:No Access User:/ nobody4:SunOS 4.x Nobody:/ >
College of Engineering: Electrical and Computer Engineering Department
Pasting files
Paste [options] file-list Options -d list use `list' characters as line separators (TAB) is default
> cat grades.txt jerry seinfield ECE 3.2 george kastanza CIS 2.5 liu hong CPE 4.0 ideas jolly ECE 2.75 drew Carey CIS 3.25 lewis stein PHY 3.5 oswald hanson MTH 3.3 jolly mon CPE 3.0 jim carey ENG 4.0 jim belushi BIS 3.5 College of Engineering: Electrical and Computer Engineering Department
Example Paste
> paste names.txt grades.txt jerry seinfield 123.111.2222 jerry seinfield george kastanza 123.222.3333 george kastanza liu hong 123.333.4444 liu hong ideas jolly 234.555.6666 ideas jolly drew Carey 234.666.7777 drew Carey lewis stein 345.777.8899 lewis stein oswald hanson 345.888.9900 oswald hanson jolly mon 456.789.0123 jolly mon jim carey 555.666.7777 jim carey jim belushi 789.123.4444 jim belushi
ECE CIS CPE ECE CIS PHY MTH CPE ENG BIS
3.2 2.5 4.0 2.75 3.25 3.5 3.3 3.0 4.0 3.5
College of Engineering: Electrical and Computer Engineering Department
Comparing File contents
To check if two files are different (e.g. versions) use DIFF
diff [options] [file1] [file2] options -b -e -h ignore trailing white space generate and display script for ed editor do fast comparison
> diff testvi.sh testvi2.sh Warning: missing newline at end of file testvi.sh 4c4 < me="your name" --> me="Paul Fortier" 15c15 < done --> done College of Engineering: Electrical and Computer Engineering Department
Comparing File contents
To check if two files are different (e.g. versions) alternative use CMP
cmp [options] [file1] [file2][offset1][offset2] options -l displays the offset and values of all mismatched bytes -s causes all output to be inhibited offsetts sets where in a file to begin comparison eng-svr-1:/home/pfortier>cat temp1 liu hong 123.333.4444 liu hong CPE 4.0 eng-svr-1:/home/pfortier>cat temp2 liu hong 123.333.4444 liu hing CPE 4.0 eng-svr-1:/home/pfortier>cmp -l temp1 temp2 30 157 151 eng-svr-1:/home/pfortier>
College of Engineering: Electrical and Computer Engineering Department
Rolling your own programs PERL (another kind of shell)
Developed in 1986 since shell not enough and C to much!! Best source: http://www.perl.com Running Perl use utility
Perl [-option] filename
Options c check for syntax, but do not execute, -v option prints out information about itself, most of the time, simply: Perl filename.pl
College of Engineering: Electrical and Computer Engineering Department
Perl Facts
Perl is a stable, cross platform programming language. It is used for mission critical projects in the public and private sectors. Perl is Open Source software, licensed under its Artistic License, or the GNU General Public License (GPL). Perl was created by Larry Wall. Perl 1.0 was released to usenet's alt.comp.sources in 1987 PC Magazine named Perl a finalist for its 1998 Technical Excellence Award in the Development Tool category. Perl is listed in the Oxford English Dictionary.
College of Engineering: Electrical and Computer Engineering Department
Perl Features
Perl takes the best features from other languages, such as C, awk, sed, sh, and BASIC, among others. Perls database integration interface (DBI) supports thirdparty databases including Oracle, Sybase, Postgres, MySQL and others. Perl works with HTML, XML, and other mark-up languages. Perl supports Unicode. Perl is Y2K compliant. Perl supports both procedural and object-oriented programming. Perl interfaces with external C/C++ libraries through XS or SWIG. Perl is extensible. There are over 500 third party modules available from the Comprehensive Perl Archive Network (CPAN). The Perl interpreter can be embedded into other systems.
College of Engineering: Electrical and Computer Engineering Department
Perl and the Web
Perl is the most popular web programming language due to its text manipulation capabilities and rapid development cycle. Perl is widely known as " the duct-tape of the Internet". Perl's CGI.pm module, part of Perl's standard distribution, makes handling HTML forms simple. Perl can handle encrypted Web data, including e-commerce transactions. Perl can be embedded into web servers to speed up processing by as much as 2000%. Mod_perl allows the Apache web server to embed a Perl interpreter. Perl's DBI package makes web-database integration easy.
College of Engineering: Electrical and Computer Engineering Department
Supported Operating Systems
Unix systems LINUX systems Macintosh - (OS 7-9 and X) Windows VMS And many more...
College of Engineering: Electrical and Computer Engineering Department
Simple Perl Program
Create a perl script #!/usr/bin/perl print "Hello, World!\n"; Run It perl -w hello.pl Or chmod 700 hello.pl ./hello.pl
College of Engineering: Electrical and Computer Engineering Department
Perl data types variables, strings and integers
Major difference between shell and perl
$ always used to denote variable, even when assigning values
$i = 3;
Integer variables support math operators+, -, *, /, =) Support range (..), strings ("st"), concatenate (.)
College of Engineering: Electrical and Computer Engineering Department
Example
eng-svr-1:/home/pfortier>cat tstperl.pl #!/usr/bin/perl print 1, 2, 3..10, "\n"; #range operator print "A", "B", "C", "\n"; #strings $i = "A". "B" ; #concatenate operator print "$i", "\n"; eng-svr-1:/home/pfortier>perl -w tstperl.pl 12345678910 ABC AB eng-svr-1:/home/pfortier>
College of Engineering: Electrical and Computer Engineering Department
Array types
An array variable is signified by the @ character. A list literal is defined by parentheses and separated by commas:
@arr_1 = (1, 2, 3); @str_arr = ("hello", "cruel", "world"); @arr_42 = qw(the meaning of life doesn\'t exist); @arr_gen = (0 .. 5);
arr_42? It is the shorthand method of writing a list of strings using the qw or quote word function
arr_gen is the same as the list (0, 1, 2, 3, 4, 5)
College of Engineering: Electrical and Computer Engineering Department
Array manipulations
Adding into an array or deleting from an array @arr_gen = (-1, @arr_gen, 6); # adds -1 to the front, 6 to the back of list ($a, @arr_gen) = @arr_gen; # removes -1 from @arr_gen to $a $length = @arr_gen; # length is 7 ($first) = @arr_gen; $arr_gen[3] = 42; @arr_gen[0,1] = (10, 11); Engineering Department College of Engineering: Electrical and Computer
More array
Another powerful feature of arrays in Perl is negative indexing:
@arr_gen = (0 .. 5); $a = $arr_gen[-2]; # $a gets assigned 4 $last = $#arr_gen; # last == 4
Last gives last the last index value of the array (not the length).
College of Engineering: Electrical and Computer Engineering Department
More array stuff
So how do we add/remove from lists with ease? Simple, treat them like stacks or queues: @arr = (1 .. 3); $new_rvalue = 4; $new_lvalue = 0; push(@arr, $new_rvalue); # @arr == (1, 2, 3, 4) unshift(@arr, $new_lvalue); # @arr == (0, 1, 2, 3, 4) $x = pop(@arr); # $x == 4, @arr == (0, 1, 2, 3) $y = shift(@arr); # $y == 0, @arr == (1, 2, 3) Instant stacks and queues. Nice! How about reversing a list? Simply call the reverse function:
@rev_a = reverse(@a); Simple enough... how about sorting a list?
@mylist = (1, 2, 4, 0, 32, 22, 17, 53, 42); @sorted = sort(@mylist); @sorted is 0, 1, 17, 2, 22, 32, 4, 42, 53. Wait! That isn't sorted... well technically it is. The sort function performs an ASCII sort, not a numeric sort.
College of Engineering: Electrical and Computer Engineering Department
Hash structures
A hash is a data structure that holds scalars, but is indexed by a key value or another scalar. Hash variables are declared with %. Elements in the hash a...