Archive

Archive for the ‘Linux’ Category

SED but true

May 15, 2015 Leave a comment

Working on a *nix like system for enough time will get you exposed to the sed command line tool. sed – which stands for “Stream Editor” – is in my opinion one of the most feared command line tools available on *nix environments, mostly since when you see someone using it – it seems like they are just typing a meaningless stream of slashes, back-slashes and quotation marks. However, this is a very powerful tool that can help you manipulate text files and streams relatively easily without much understanding of regular expressions.
The most basic capability in sed is the ability to search and replace values – and the syntax is rather simple: sed 's/value to search for/value to put instead/g'.
The s at the beginning is mandatory (for search and replace) – think of it as saying “Search-and-replace”. The g at the end is optional and stands for “Global” – it tells sed whether to replace all matches in a line or just the first match (in a line). Here are some examples of simple find and replace scenarios.

~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/'
Hello sir, this indeed a confusing example
~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/g'
Hello sir, this indeir a confusing example

Usually, you would need some understanding of regular expressions (regex) to do advanced operations:

~> echo 'Hello sir, this indeed a confusing example' | sed 's/sir.*/world/'
Hello world

In the above example we tell sed to find the text ‘sir’ with zero or more characters after it – and replace these with the text ‘world’. The designation ‘.*’ in regex means zero or more characters of any kind (letters, punctuation, numbers, white spaces etc).

Regex Groups

A useful (yet confusing) ability related to regular expressions is support of groups. Groups in regex essentially mean matches of the regular expression – so if we take the text “A=B”, and apply the regular expression: “(.*)=(.*)”, we will get two groups the first with the value “A” and the second with the value “B”. Unfortunately – sed doesn’t want us to have a simple life – so “(.*)=(.*)” is actually “\(.*\)=\(.*\)”. Once I matched these groups – I can use them in the “value to put instead” part of the sed command – they can be referred to as \x – where x is the index of the group. so \1 is the first group, and so on. Example:

~> echo "A=B" | sed 's/\(.*\)=\(.*\)/If \1=\2, I can assert that \2=\1 as well/'
If A=B, I can assert that B=A as well
Useful sed Patterns

Here are some patterns I found useful:

~> echo "Everything in this line is a secret, apart from this" | sed 's/.*from //'
this
~> echo 'I would like to thank my kids, my mother, my boss, my neighbors and most importantly - my wife!' | sed 's/ and most.*/./'
I would like to thank my kids, my mother, my boss, my neighbors.
~> echo "A1B2C3D4" | sed 's/[0-9]*//g'
ABCD
~> echo "My phone number is: +1-212-555-9876" | sed 's/[^0-9\+\-]*//g'
+1-212-555-9876
Advanced Usages

Lacking a better way to express it – I can say that sed has crazy capabilities. Apart from inserting lines, deleting lines it can even run shell commands as part of its work.

I recently wrote myself and ad-hoc template engine based on sed – and this is the entire code:

cp template target

while read line; do
   KEY=$(echo $line | cut -d"=" -f1)
   VALUE=$(echo $line | cut -d"=" -f2)
   echo replacing '$('${KEY}')' with $VALUE
   sed -i '' "s/\$(${KEY})/${VALUE}/g" target
done < key_value_input_file

You can see a working example of the code here: http://ideone.com/ISxTMj

References

Bash Epoch Converter

July 10, 2014 Leave a comment

I use the online epoch converter on an almost daily basis with an almost single usage – converting a value of seconds/milliseconds since the epoch (1/1/1970 00:00:00) to a human readable date.
I wanted something quicker which I can use from my Terminal window without – so I created a Bash function that performs that:

epoc usage

epoc usage


The function

epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')"
}
epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')"
}

One liners:

# UNIX, Mac OSx and FreeBSD
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" ; }
# Linux
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" ; }
%d bloggers like this: