i0S8F

            digit      numbrimärk   цифра
            quantity   arv          число
            number     number       номер

Learning AWK*
-------------

*) It is not an abbreviation for awkward

@I started to read excellent tutorial

  [0]. http://www.grymoire.com/Unix/Awk.html

to do this. So, here are my notes. This is conspect, so the main source is not referenced at all, instead of it my own impressions are embraced into 'at' 'backslash''at' pair. I also use other sources, which are referenced as usual [1]

  [1]. https://www.tutorialspoint.com/awk/index.htm

Create test file

  $ echo \
  > null üks kaks kolm neli viis kuus seitse kaheksa üheksa | \
  > tr ' ' '\n' > num
  $ cat num
  null
  üks
  ..
  üheksa

  $ $ cat num | \
  > awk 'BEGIN {print "ALGUS"} {print} END {print "LÕPP"}'
  ALGUS
  null
  ..
  üheksa
  LÕPP

Why AWK (Aho-Weinberger-Kernighan)?

Init and allocate, loop, free allocated resources and destroy:
it seems to be the most general patternin many areas.\@

  $ cat num | \
  > awk '
  > BEGIN {print "ALGUS"}
  >       {print " ", $0}
  > END   {print "LÕPP" }
  > '
  ALGUS
    null
    ..
    üheksa
  LÕPP

NOTE, that
----

  "  " $0

is concatenation, but

  " ",$0

is enumeration with default separator ' ' (blank, symbol you usualy put between the words, built-in variable OFS Output Field Separator)

and

  "$8 $3"

doesn't mean

  $8, $3

Important Concept
.................

As one may see all those shabangs and qoutes are for sh.

  #!/bin/awk -f
  awk '..'

So, lets say Hello now.

  $ cat buu.awk
  { print "Hello" }
  $ echo buu | awk -f buu.awk
  Hello

Constructing string with sh

  $ ZWEI=2
  $ echo one two three | awk '{ print $'$ZWEI' }'
  two

However

  $ echo one two three | awk '{ print $ZWEI }'
  awk: illegal field $(), name "ZWEI"
   input record number 1, file
   source line number 1

Note:  In the C and Bourne shell, the quote is just a switch. It turns the interpretation mode on or off. There is really no such concept as "start of string" and "end of string". The quotes toggle a switch inside the interpretor.

The Bourne shell has a mechanism to provide a value for a variable if the value isn't set, or is set and the value is an empty string. This is done by using the format:

  ${variable:-defaultvalue}

That's why

  $ echo one two three | awk '{ print $'${DREI:-3}' }'
  three

The format of the original AWK is not free-form. You cannot put new line breaks just anywhere. They must go in particular locations. To be precise, in the original AWK you can insert a new line character after the curly braces, and at the end of a command, but not elsewhere. If you wanted to break a long line into two lines at any other place, you had to use a backslash. Try to include an AWK or sed script within a C shell script, and the back slashes will drive you crazy.

The parenthesis and semicolon are optional.

Arrays
......

Thomas Weidenfeller
1/22/09
bolta...@yahoo.co.uk wrote:

  https://groups.google.com/forum/#!topic/comp.lang.awk/KI7STwHdb40

> Is there an easy way to
> initialise it in one go in the same vein as this C example, eg:
>
>   char *arr[] = { "hello" , "cruel", "world" };
> ..

The typical hack is to use split().

/Thomas

Selftest. my first nice table in awk
....................................

        It can be considered to be a pseudo-C interpretor,
        as it understands the same arithmatic operators as C.

#!/bin/sh

awk '
  function framed(a) {
    print    "+--------------------------------------------+"
    for (i=1; i<=5; i++) printf("|%s\t", a[i])
    print "|\n+--------------------------------------------+"
    for (i=6; i<=10; i++) printf("|%s\t", a[i])
    print "|\n+--------------------------------------------+\n"
  }
  BEGIN {
    split("zero one two three four five six seven eight nine",\
          a, " ")
    framed(a)
    split("null üks kaks kolm neli viis kuus seitse kaheksa üheksa",\
          a, " ")
    framed(a);
    split("null eins zwei drei vier fünf sechs sieben acht neun",\
          a, " ")
    framed(a)
  }
' | expand -t 9

Not what i expected. Reminds me Ruby learning curve. Much help from

  https://github.com/freebsd/freebsd/blob/master/sys/tools/usbdevs2h.awk

## STANDARD AWK VARIABLES [1]

  1) awk 'BEGIN { print ENVIRON["USER"] }'

  To find names of other environment variables, use env command.

  2) awk 'BEGIN {print "FS = " FS}' | cat -vte

  man cat show tabs as ^I end as $

  3) echo "One Two
     One Two Three
     One Two Three Four" | \
     awk 'NF > 2'

  4) RLENGTH represents the length of the string
     matched by match function.

  Testing with FreeBSD 1.1.1

     $ awk '
         BEGIN { if (match("Üks Kaks Kolm", "[üa-z]+"))
         { print RLENGTH } }
     '
     1
     awk 'BEGIN {if (match("One Two Three"," [a-z]+ ")) print RLENGTH}'
     5

  @ FreeBSD 10+ is not unicode friendly, as somebody claimed.
    Motivation grounded -- Heirloom Toolchest?

  Ubuntu Bionic:
  $ cd ~/heirloom-070715/nawk
  $ awk 'BEGIN {if (match("One Üks Ein"," [a-zÜ]+ ")) print RLENGTH}'
  6
  $ ./awk 'BEGIN {if (match("One Üks Ein"," [a-zÜ]+ ")) print RLENGTH}'
  5

FreeBSD: (https://forums.freebsd.org/threads/14908/) FreeBSD has it's own libc implementation. And yes, it's different from glibc which is primarily a linux thing.

Build:

Install dependencies:

  # pkg install wget flex bison ncurses

Add missing header as /usr/include/utmp.h (found here:)

  http://www.retro11.de/ouxr/211bsd/usr/include/utmp.h.html

Code:

  #ifndef _UTMP_H_
  #define _UTMP_H_

  #define _PATH_UTMP  "/var/run/utmp"
  #define _PATH_WTMP  "/usr/adm/wtmp"

  #define UT_NAMESIZE 15
  #define UT_LINESIZE 8
  #define UT_HOSTSIZE 16
  struct utmp {
      char    ut_line[UT_LINESIZE];
      char    ut_name[UT_NAMESIZE];
      char    ut_host[UT_HOSTSIZE];
      long    ut_time;
  };

  #endif

Build libraries:

  $ wget -c --no-check-certificate https://freefr.dl.sourceforge.net
      /project/heirloom/heirloom/070715/heirloom-070715.tar.bz2
  $ tar xf heirloom-070715.tar.bz2

  $ cd /usr/home/virtu/heirloom-070715
  $ make
  $ cd nawk

Patch nawk with diff below and build it:

  $ diff main.c.orig main.c
  69c69
  < int main(int argc, unsigned char *argv[], unsigned char *envp[])
  ---
  > int main(int argc, char *argv[], char *envp[])
  $ make

Use:

  $ ./awk 'BEGIN {if (match("One Üks Ein"," [a-zÜ]+ ")) print RLENGTH}'
  5

Going Perl 1978 -> 1987
.......................

Ah, it's so Perlish!

  $ cat perlish.pl
  #!/usr/local/bin/perl

  $sentence="The river flows slowly.";
  if($sentence =~ /river/)
      {
      print "Matched river.\n";
      }
  else
      {
      print "Did not match river.\n";
      }

  $ awk 'BEGIN { print "The river flows slowly." ~ "river" ? \
   "Matched river." : "Did not match river." }'

Now this:

  https://stackoverflow.com/questions/627661
  /how-can-i-output-utf-8-from-perl

Thanks, finally got an solution to not put utf8::encode all over code. To synthesize and complete for other cases, like write and read files in utf8 and also works with LoadFile of an YAML file in utf8

  use utf8;
  use open ':encoding(utf8)';
  binmode(STDOUT, ":utf8");

  open(FH, ">test.txt");
  print FH "something éá";

  use YAML qw(LoadFile Dump);
  my $PUBS = LoadFile("cache.yaml");
  my $f = "2917";
  my $ref = $PUBS->{$f};
  print "$f \"".$ref->{name}."\" ". $ref->{primary_uri}." ";

where cache.yaml is:

  ---
  2917:
    id: 2917
    name: Semanário
    primary_uri: 2917.xml

See also "Why does modern Perl avoid UTF-8 by default?"

Going Python (1994)..

Annex A. Characters, Symbols and the Unicode Miracle -- Computerphile
---------------------------------------------------------------------

I will leave a youtube link here, featuring Tom Scott's Characters, Symbols, Unicode miracle: goo.gl/sUr1Hf. You get to hear and see how everything's being evolved from ASCII character encoding to utf-8. – Roylee Dec 24 '15 at 11:36

  https://stackoverflow.com/questions/9533258
    /what-is-the-maximum-number-of-bytes-for-a-utf-8-encoded-character

youtube Transcript

  https://www.youtube.com/watch?v=MijmeoH9LT4

utf-8 is perhaps the best hack, the best single thing that's used that can be written down on the back of a napkin and that's how it was put together it was the first draft of utf-8 was written on the back of a napkin in a diner.

And it's just such an elegant hack that solves so many problems that i absolutely love it. Back in the 1960s we had teleprinters we had simple devices that you type a key and it sends some numbers and the same letter comes out the other side.

But there needs to be a standard, so in the mid-1960s America at least settled on ASCII which is the American Standard Code for information interchange, and it's a seven bit binary system so each letter you type in gets converted into seven binary numbers and sent over the wire. Now that means you can have numbers from 0 to 127 they moved the first 32 for control codes and less important stuff for writing. Things like 'go down a line' or 'backspace' and then they made the rest characters -- added some numbers and punctuation marks. They did a really clever thing which is that they made A 65 which in binary is 1000(01)

       64 32 16  8  4  2  1
        0  0  0  0  0  0  0  =   0
        1  1  1  1  1  1  1  = 127
  'A'      1  0  0  0  0  1  =  65

which means that B is 66 which means you've got two in binary just here
1000(10), C 67 1000(11) 3 in binary so you can look at a 7 bit binary character and just note at the first two digits and know what its position in the alphabet is. Even cleverer than that they started lowercase 32 later which means that lowercase 'a' is 97 11000(01) anything that doesn't fit into that is probably a space which conveniently will be all zeroes or some kind of punctuation mark.

Brilliant, clever, wonderful, great way of doing things and that became the standard at least in the english-speaking world.

as for the rest of the world they, a few of them, did versions of that but you start getting into other alphabets, into language that don't really using alphabets at all, they all came up with their own encoding which is fine.. and then along come computers and over time things changed we moved to 8-bit computers so we now have a whole extra number at the start just to confuse matters, which means we can go to 256 and we can have twice as many characters and of course everyone settled on the same standard for this because that would make it perfect (well, none of them did). All the Nordic countries start putting Norwegian characters and Finnish characters in there. Japan just doesn't use ASCII at all. Japan goes and creates its own multibyte encoding with more letters and more characters more binary numbers going into individual character.

All these things are massively incompatible. Japan actually has three or four different encodings, all of which are completely incompatible with each other so you send a document from one old-school Japanese computer to another -- it will come out so garbled that there is even a word in Japanese for garbled characters which is, I'm probably mispronouncing this but it's 'mogi bark' a bit of a nightmare, but it's not bad because how often does someone in London have to send a document to a completely incompatible an unknown computer at another company in Japan. In those days it's rare -- you print it off and you faxed it.

And then the the world wide web hit and we have a problem because suddenly documents being sent from all around the world, all the time. So a thing is set up called the Unicode consortium.

I mean what I can only describe as a miracle. Over the last couple of decades they have hammered out a standard Unicode now have a list of more than a hundred thousand characters that covers everything you could possibly want to write in any language -- English alphabet, Cyrillic alphabet, Arabic alphabet, Japanese, Chinese and Korean characters. What you have at the end is the Unicode consortium assigning a hundred thousand plus characters to a hundred thousand numbers. They have not chosen binary digits, they've not chosen what should be represented. Fs all the other side is that Arabic character there (that is number 5700.. something) and this linguistic symbol here (that's ten thousand.. something).. I have to simplify massively here because they're about of course five or six incompatible ways to do this.

But what the web has more or less settled on is something called utf-8.

There are a couple of problems with doing the obvious thing which is saying: okay we're going to a hundred thousand that's going to need (to be safe, that's going to need) thirty two binary digits to encode it.

They encoded the English alphabet in exactly the same way as ASCII did. 'A' is still sixty five so if you have just a string of English text and you're encoding it at 32 bits per character you're gonna have about 26 27 zeros and then a few ones for every single character. That is incredibly wasteful.

Suddenly every english-language text file takes four times the space on disk. So problem one -- you have to get rid of all the zeros in English text. Problem two -- there are lots of old computer systems that interpret eight zeros in a row and null as this is the end of the string of characters. So if you ever send eight zeros in a row they just stop listening, they assume that the string has ended there and it gets cut off. So you can't have eight zeros in a row anywhere.

Problem number three -- it has to be backwards compatible you have to be able to take this Unicode text and chuck it into something that only understands basic ASCII and have it more or less work for English text.

utf-8 solves all these problems and it's just a wonderful hack it starts by just taking asking: if you have something under 128 but can just be expressed as seven digits you put down a zero and then you put the same numbers that you would otherwise.

So let's have that 'A' again:

  UTF-8

  01100001  A  65

That is still UTF-8-valid. And that is still ASCII-valid. Brilliant. Okay.

Now let's say we're going above that, now you need something that's going to work more or less for us, or at least not break things, but still be understood. So what you do -- is you start by writing down

  110

This means (this is the start of a new character and this character is going to be 2 bytes long) two ones -- two bytes, a byte being 8 characters, and you're saying this one I'm going to start it with 'one' 'zero', which means this is a continuation and then all these blank spaces of which you have five here and six here you fill-in the other numbers.

  110xxxxx 10xxxxxx

And then when you're calculating it you just take off those headers and it understands just these

  ___00110 __110010  <00110110010 = 434 btw!>

as being whatever number that turns out to be that's probably somewhere in this hundreds.

That'll do you for the first 4096. What about above that? All above that you go 'one' 'one' 'one' 'zero' meaning there are three bytes in this three ones three bytes with two continuation bytes.

  1110xxxx 10xxxxxx 10xxxxxx

So now you have sixteen spaces. You want to go above that --you can. The specification goes all the way to 1111110x with this many continuation bytes after it:

  1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

It's a neat hack that you can explain on the back of a napkin or a bit of paper, it's backwards compatible, it avoids waste, at no point will it ever ever ever send eight zeros in a row, and really really crucially the one that made this win over every other system, is that you can move backwards and forwards really easily. You do not have to have an index where the characters start -- if you are halfway through a string and you want to go back one character you just look for the previous header.

And that's it and that worked and as of a few years ago utf-8 beat out ASCII and everything else as for the first time the dominant character encoding on the web.

We don't have that "Mogi barking" that Japanese has, we have something that nearly works, and that is why it's the most beautiful hack that I can think of those used around the world every second and every day.

why tom said utf-8 only 'nearly' works?

..an old system that has been programmed well will take those nice curly quotes* Microsoft Word is put into Unicode, and it will look at that and say -- that is three separate characters!

  <computerphile>
  facebook.com/computerphile @computer_phile

------
*) “Hello” e2 80 9c 48 65 6c 6c 6f e2 80 9d
   microsoft powershell ..cat vs unix cat:)

  PS G:\tmp> cat .\hello.word.txt | od -x
  0000000 483f 6c65 6f6c 0d3f 000a
  0000011
  PS G:\tmp> bash
  $ cat hello.word.txt | od -x
  0000000 bbef e2bf 9c80 6548 6c6c e26f 9d80
  0000016

Annex B. Using AWK to Process Input from Multiple Files
.......................................................

  https://stackoverflow.com/questions/14984340
    /using-awk-to-process-input-from-multiple-files

..What follows is the answer I was looking for (and that I think most people would be), i.e., simply to concatenate every line from two different files using AWK. Though you could probably use some UNIX utilities like join or paste, AWK is obviously much more flexible and powerful if your desired output is different, by using if statements, or altering the OFS (which may be more difficult to do depending on the utility; see below) for example, altering the output in a much more expressive way (an important consideration for shell scripters.)

For simple line-by-line concatenation:

  awk 'FNR==NR { a[FNR""] = $0; next }
               { print a[FNR""], $0 }' file1 file2

This emulates the function of a numerically indexed array (AWK only has associative arrays) by using implicit type conversion. It is relatively expressive and easy to understand.

Using two files called test1 and test2 with the following lines:

file1:

  line one
  line two
  line three

file2:

  line four
  line five
  line six

I get this result:

  line one line four
  line two line five
  line three line six

Depending on how you want to join the values between the columns in the output, you can pick the appropriate output field separator. Here's an example with ellipses (...) separating the columns:

  awk 'BEGIN   { OFS="..."}
       FNR==NR { a[(FNR"")] = $0; next }
               { print a[(FNR"")], $0 }' test1 test2

Yielding this result:

line one...line four
line two...line five
line three...line six

I hope at least that this inspires you all to take advantage of the power of AWK!

The raise and fall of ARGIND
----------------------------

colors

  Red
  Orange
  Yellow
  Green
  Blue
  Indigo
  Violet

farben

  Rot
  Orange
  Gelb
  Grün
  Blau
  Violett

  $ awk '{ print ARGIND, FNR, NR, $0 }' colors farben

  1 1 1 Red
  1 2 2 Orange
  1 3 3 Yellow
  1 4 4 Green
  1 5 5 Blue
  1 6 6 Indigo
  1 7 7 Violet
  2 1 8 Rot
  2 2 9 Orange
  2 3 10 Gelb
  2 4 11 Grün
  2 5 12 Blau
  2 6 13 Violett

If you have gawk, just test ARGIND*:

  awk '
  ARGIND == 1 { do file 1 stuff; next }
  ARGIND == 2 { do file 2 stuff; next }
  ' file1 file2

In other awks though you can just test for the file name:

  awk '
  FILENAME == ARGV[1] { do file 1 stuff; next }
  FILENAME == ARGV[2] { do file 2 stuff; next }
  ' file1 file2

------
*) Found here:

      https://stackoverflow.com/questions/24739570
        /how-to-handle-3-files-with-awk