awk

It is slippery, simple, small and brilliant. No programming language, with the exception of Brainfuck, has managed to get me this excited in such a short time. The possibilities this little line processor offers are enormous. Its range of work goes from small file edits à la sed, through improved Unix commands like cat and grep and merging two CSV files, all the way to a complete database.
I am so taken with the language that I have decided, without further ado, to implement the next two projects at work that involve data mining and/or data merging in awk. The development time is so short that with awk one can genuinely speak of RAD (Rapid Application Development).
Since my weblog has lately resembled a link collection more than anything else, and I know that many programmers read my blog, or at least people who are enthusiastic about technology (I won’t say geeks), I am going to show a few examples in awk.
First of all, anyone who wants to follow the examples needs awk. awk really is available for every platform. Although its roots are in the eighties, i.e. in Unix times, there is awk for DOS/Windows too. All other operating systems usually ship with awk out of the box. I for my part use mawk 1.3.3. SuSE Linux has gawk AFAIK, but the two are largely compatible with my examples.
As I mentioned above, awk is a line interpreter. That is, lines are read from a CSV file and split into fields. These fields can be manipulated and then printed. A small awk program that mimics cat looks like this.

{
  print $0
}

$0 stands for a complete line of the input (CSV file or regular file). The curly braces can be seen as a kind of function that is called for every line.
Simplified, calling an awk program looks like this.

lemming@ibook:~ $ awk '{ print $0; }'

The program code is on the command line and the data is read from STDIO.

lemming@ibook:~ $ ls -l | awk '{ print $0; }'

The data can also be read from a file.

lemming@ibook:~ $ awk '{ print $0; }' daten.csv

You can also store a program in a file.

lemming@ibook:~ $ awk -f meincat.awk daten.csv

The whole thing gets interesting as soon as you manipulate the fields of a CSV input.
For the following examples I’ll use this tab-separated table, an excerpt from my former music site lemvista.de.

id	Interpret	Titel	Abrufe
1	The Cardigans	My Favorite Game	2470
2	Metallica	Die Die Die My Darling	730
3	Eric Clapton	Cocaine	230
4	Eminem	8 Mile	1571

The awk code to display only the title (Titel) with the artist (Interpret) in front of it would look like this:

{
  print $2 " - " $3;
}

lemming@ibook:~ $ awk -v FS="\t" '{ print $2 " - " $3; }' songs.tsv
Interpret - Titel
The Cardigans - My Favorite Game
Metallica - Die Die Die My Darling
Eric Clapton - Cocaine
Eminem - 8 Mile

In addition, the processing of a block (the {} braces) can be tied to a RegExp condition. That makes sense when you use several blocks. FS is the variable that specifies the field separator, in our case the tab. As seen above, variables can also be changed from the command line.

BEGIN { FS ="\t";
}
/Clapton/{
  #Dieser Block wird nur ausgeführt,
  #wenn Clapton in der Zeile vorkommt
  $2 = "Slowhand"
}
/Eminem/{
  #Dieser Block wird nur ausgeführt,
  #wenn Eminem in der Zeile vorkommt
  $2 = "Slim Shady"
}
{
  print $2 " - " $3;
}

The Cardigans - My Favorite Game
Metallica - Die Die Die My Darling
Slowhand - Cocaine
Slim Shady - 8 Mile

Besides the blocks that process lines, there is also a BEGIN and an END block which, as the names suggest, are executed at the beginning and at the end of a program.

BEGIN{ print "Abrufe werden zusammen gezählt";
FS = "\t";
}
{
  #Aufrufe zusammen zählen
  if (i > 0) { #Die erste Zeile wird ignoriert.
    abrufe = abrufe + $4;
  }
  i++;
}
END {
  print i " Titel wurden " abrufe "x abgerufen.";
}

lemming@ibook:~ $ awk -f count_loads.awk songs.tsv
Abrufe werden zusammen gezählt
5 Titel wurden 5001x abgerufen.

In the second part of my little awk tutorial I will merge two CSV files with each other. If you have questions about awk, just ask them in the comments.