Sunday, 27 April 2014


I recently discovered a variable in BASH which is going to stick as one of my favourite variables (everybody has one).
That is the $RANDOM variable.

It generates a pseudorandom number between 0 and 32767. This is an awkward choice of limits, but it can be adjusted by just taking a modulus:

$ echo $((RANDOM%2))

outputs 0 or 1, or:

$ echo $((RANDOM%10))

outputs a number between 0 and 9.

This inspired me to do some simple calculations on random numbers which you can find in my other blog post.
If you want the details of what and how I did them (apologies if most of it is just quick and dirty scripting), keep reading!

I produced the random distribution of points in 2D, by creating two lists of numbers (doubles) between 0 and 1, joining them and use gnuplot to plot them:

$ for i in {1..5000} ; do echo "$((RANDOM%10000)) / 10000" | bc -l; done > random.txt
$ sort --random-sort random.txt > random2.txt
$ paste random.txt random2.txt > randomjoin.txt
gnuplot> plot 'randomjoin.txt' using 1:2 pt 7 ps 0.4 notitle

Funnily enough, the uniform distribution - which I generated by applying a small "wobble" to a grid of evenly spaced points - was more complicated to produce:

$ for i in {0..70} ; do for j in {0..70}; do echo "$i $j" ; done ; done | awk '{print $1/70 " " $2/70}' > uniform.txt
$ sed 's/\./\.00/g' random.txt > wobble1.txt
$ sed 's/\./\.00/g' random2.txt > wobble2.txt
$ paste wobble1.txt wobble2.txt > wobblejoin.txt
$ paste uniform.txt wobblejoin.txt > uniformjoin.txt
$ awk '{print $1-$3 " " $2-$4}' uniformjoin.txt > uniformwobble.txt
gnuplot> plot 'uniformwobble.txt' using 1:2 pt 7 ps 0.4 notitle

The 1D plots have been generated with the same data and some re-binning tricks in gnuplot:

gnuplot> binwidth=0.1
gnuplot> bin(x,width)=width*floor(x/width)
gnuplot> plot 'random.txt' using (bin($1,binwidth)):(1.0) smooth freq with boxes notitle

For the coin tosses example, I generated the sequences using some BASH:

$ for i in {1..100000} ; do for i in {1..15}; do echo -n $((RANDOM%2)); done | sed 's/0/H/g' | sed 's/1/T/g' && echo; done > tosses.txt

And analyzed, counting the repetition (or better, the non-repetitions) in it with:

$ echo "100000 - `grep -v 'HHH' tosses.txt | grep -v 'TTT' | wc -l`" | bc -l

As a bonus, the same set can also be used to explore the number of head (or tails) for each occurrence. Given that for a fair coin there is a 50% chance of getting head (and tails), the result should be having about half heads and half tails for most cases. Below are listed the number of instances (first column) for a given number of heads (second column):

$ while read line ; do echo $line | grep -o H | wc -l ; done < tosses.txt > freq.txt
$ more freq.txt | sort | uniq -c

      3       0
     51      1
    312     2
   1359    3
   4189    4
   9105    5
  15338   6
  19608   7
  19806   8
  15175   9
   9145    10
   4095    11
   1393    12
    362     13
     55      14
      4       15

This is a binomial distribution centered around 7.5, as expected.

Monday, 18 November 2013

Use sed to retreive links from a Firefox bookmark file

Sed is a great text manipulation tool. It allows to manipulate text through simple substitution commands.

It did occur to me that I had a few backup .json files in which Firefox bookmarks were stored. As I did not want to overwrite the existing bookmarks, but I wanted to retreive just a few links, I tried to open the file with a text editor.

The result was not pleasant:

I decided to use some CLI magic to transform this in a list of URLs which is easily readable, instead. It can be done in one line:

$ sed "s/\"uri\"\:\"/\\`echo -e '\n\r'`/g" bookmarks.json | sed "s/\"[^@]*$//" > new_bookmarks.txt

Where:   bookmarks.json -> bookmark file name
            new_bookmarks.txt -> new file name where the links will be saved


Thursday, 30 May 2013

MailMe: how to get notified automatically via email through BASH

In my job, I have to perform calculations on a remote machine to which I connect through ssh. It happens quite often that I have to run a few jobs overnight, if not for days. I could check manually for their termination, but I came up with a better way.

In a previous post I described how useful it can be to run script in background on the machine without having to be logged-in. Combining the power of screen with the utility of mail we can create a script which will automatically send an email whenever the job is finished.

The Script

## This scripts sends an automated email as soon as a job is finished
## ***   Use with screen:
## ***  
## ***   $ screen -S [job-id]               (start screen)
## ***   $ mailme [job-id]                  (run mailme in screen)
## ***   Ctrl+a d                           (detach)
## ***   Ctrl+d                             (kill screen, from session)
## ***   $ screen -ls                       (check running screens)
## ***   $ screen -r [screen-id]            (resumes session)
## ***   screen -X -S [screen-id] kill      (kill screen, from outside)
## ***  

# Need argument: job id which needs to be checked for completion
if [ -z "${1}" ] ; then
    echo "Please insert job ID as argument"
    echo "Exiting status 1"

    exit 1

# Need second argument: email
if [ -z "${2}" ] ; then
    echo "Please insert email as second argument"

    echo "Exiting status 1"
    exit 1

# Put your email here
zzFREQ=3600 #Frequency of checks, in ms

# Loop that checks job status
while "${zzEND}"

    zzSTRING=$(bhist -d | grep "${zzJOBID}")
    if [ ! -z "${zzSTRING}" ] ; then
    echo -e "Dear Sir,\n\nThis is an automatic email from grep Linux indicating that the job ID\# ${zzJOBID} has been terminated (succesfully or not).\n\nThis is the output:\n\n $(bhist -d) \n\nBest, \ngrep Linux | mail -s "Job ${zzJOBID} Completed!" "${zzEMAIL}"
    sleep "${zzFREQ}"

exit 0 

The comments should provide a satisfactory explanation, but it could be used straight away for jobs submitted through the routine bsub.

You would just need to run the script from a screen and the game is on. If you do not remember how to do this, I have put some reminders in the header section of the script, or you can check my guide to screen.

You can run multiple of these scripts in multiple screen sessions if more jobs need to be tracked.


This script has been designed to check job progress and send an email  whenever the job is complete. Through an easy modification of the condition to be checked in the variable zzSTRING in the loop, anything can trigger the delivery of an automated email, potentially making the scope of this script very broad.

Have a play with it and let me know how you use/modify it!