Archive

Archive for the ‘Shell Scripting’ Category

SED but true

May 15, 2015 Leave a comment

Working on a *nix like system for enough time will get you exposed to the sed command line tool. sed – which stands for “Stream Editor” – is in my opinion one of the most feared command line tools available on *nix environments, mostly since when you see someone using it – it seems like they are just typing a meaningless stream of slashes, back-slashes and quotation marks. However, this is a very powerful tool that can help you manipulate text files and streams relatively easily without much understanding of regular expressions.
The most basic capability in sed is the ability to search and replace values – and the syntax is rather simple: sed 's/value to search for/value to put instead/g'.
The s at the beginning is mandatory (for search and replace) – think of it as saying “Search-and-replace”. The g at the end is optional and stands for “Global” – it tells sed whether to replace all matches in a line or just the first match (in a line). Here are some examples of simple find and replace scenarios.

~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/'
Hello sir, this indeed a confusing example
~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/g'
Hello sir, this indeir a confusing example

Usually, you would need some understanding of regular expressions (regex) to do advanced operations:

~> echo 'Hello sir, this indeed a confusing example' | sed 's/sir.*/world/'
Hello world

In the above example we tell sed to find the text ‘sir’ with zero or more characters after it – and replace these with the text ‘world’. The designation ‘.*’ in regex means zero or more characters of any kind (letters, punctuation, numbers, white spaces etc).

Regex Groups

A useful (yet confusing) ability related to regular expressions is support of groups. Groups in regex essentially mean matches of the regular expression – so if we take the text “A=B”, and apply the regular expression: “(.*)=(.*)”, we will get two groups the first with the value “A” and the second with the value “B”. Unfortunately – sed doesn’t want us to have a simple life – so “(.*)=(.*)” is actually “\(.*\)=\(.*\)”. Once I matched these groups – I can use them in the “value to put instead” part of the sed command – they can be referred to as \x – where x is the index of the group. so \1 is the first group, and so on. Example:

~> echo "A=B" | sed 's/\(.*\)=\(.*\)/If \1=\2, I can assert that \2=\1 as well/'
If A=B, I can assert that B=A as well
Useful sed Patterns

Here are some patterns I found useful:

~> echo "Everything in this line is a secret, apart from this" | sed 's/.*from //'
this
~> echo 'I would like to thank my kids, my mother, my boss, my neighbors and most importantly - my wife!' | sed 's/ and most.*/./'
I would like to thank my kids, my mother, my boss, my neighbors.
~> echo "A1B2C3D4" | sed 's/[0-9]*//g'
ABCD
~> echo "My phone number is: +1-212-555-9876" | sed 's/[^0-9\+\-]*//g'
+1-212-555-9876
Advanced Usages

Lacking a better way to express it – I can say that sed has crazy capabilities. Apart from inserting lines, deleting lines it can even run shell commands as part of its work.

I recently wrote myself and ad-hoc template engine based on sed – and this is the entire code:

cp template target

while read line; do
   KEY=$(echo $line | cut -d"=" -f1)
   VALUE=$(echo $line | cut -d"=" -f2)
   echo replacing '$('${KEY}')' with $VALUE
   sed -i '' "s/\$(${KEY})/${VALUE}/g" target
done < key_value_input_file

You can see a working example of the code here: http://ideone.com/ISxTMj

References
Advertisements

Bash Epoch Converter

July 10, 2014 Leave a comment

I use the online epoch converter on an almost daily basis with an almost single usage – converting a value of seconds/milliseconds since the epoch (1/1/1970 00:00:00) to a human readable date.
I wanted something quicker which I can use from my Terminal window without – so I created a Bash function that performs that:

epoc usage

epoc usage


The function

epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')"
}
epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')"
}

One liners:

# UNIX, Mac OSx and FreeBSD
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" ; }
# Linux
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" ; }

Personalizing your connection to QA/Production environments with SecureCRT

January 6, 2014 Leave a comment

The trouble with production/QA environments is that you can rarely change them to fit you personal preferences, and usually you use a common user to access them.
If you are working on Windows and using SecureCRT to connect to Linux/Unix machines (and if not – ask yourself why …) you can personalize any connection to QA/Production environments by telling SecureCRT to perform automatic login activities – this without effecting how the machine behaves for other users.
By personalization I am referring to the definition of your personal aliases etc.

How to do it:

  1. Download this VBScript to your computer, and place it somewhere (e.g. C:\Utils\SecureCRT\Scripts)
  2. Find a connection you want to effect in SecureCRT and open it’s properties
  3. Navigate to Connection -> Logon Actions
  4. Check the Logon script checkbox and type the path to the file defined at the beginning (e.g. C:\Utils\SecureCRT\Scripts\EnvironmentPersonalizer.vbs)

Every time you login – SecureCRT will run that script.

What the script does:
The way this scripts works is that it waits for something to happen on the screen and then sends commands.
Example:

objTab.Screen.WaitForString("$")   // Wait for the prompt
objTab.Screen.Send "bash" & vbcr   // Switch to a BASH shell
objTab.Screen.Send "clear" & vbcr   // clear the screen
objTab.Screen.Send "alias ll='ls -lart'" & vbcr   // Alias 'll' to be 'ls –lart' 

You can find a few more utility scripts on my Github Repository – https://github.com/ronkitay/SecureCRT-Environment-Personalization

Note: Take great care when effecting anything on production environments – don’t do anything that will effect how the environment behaves for other people (like changing the .bash_profile for example)

Categories: Shell Scripting, Tools, UNIX Tags:

Automatically setting permanent Environment Variables on Windows

August 26, 2013 Leave a comment

More than once I needed to define a permanent Environment Variable on a Windows machine – mostly as part of some automatic installation process. E.g. – like the Java installer sets the JAVA_HOME during installation.

Windows has a very simple utility to do so – SETX.

# Set the variable HELLO with the value "WORLD" for the current user
SETX HELLO WORLD
# Set the variable HELLO with the value "WORLD" for all users (System)
SETX HELLO WORLD /m

Note that SETX is different from SET – invoking SETX as part of a shell script will not make the variable available for that shell script.

Reading material
Official Documentation – http://technet.microsoft.com/en-us/library/cc755104.aspx
Non-Official Documentation – http://ss64.com/nt/setx.html

Jar scanning utility – find a class within all jars (Unix/Linux)

March 28, 2013 Leave a comment

Every so often I need to figure out which jar(s) contain a certain class so I can figure out why an application acts the way it does – either since a class is missing in some environment or since the wrong class was loaded into the classpath.

When that happens, I use the following script. I saw it once in a company I worked at, and then again in another company.
I modified it slightly, but the credits do not go to me, they go to the anonymous person who wrote the initial code.

The script is available for download on GitHub.

Or you can simply copy it from here:

#!/bin/tcsh -f

if ( $#argv < 2 ) then
        printf "Scans all the JARs in the specified directory for classes matching the specified string\n"
        printf "Usage: $0 <dir> <class-name-pattern>\n"
        printf "Example: $0 . Exception\n"
        printf 'Example: '$0' $JAVA_HOME/lib "com\/sun\/.*action"\n'
        exit 1
endif

set DIR = "$1"
set CLASS = "$2"

foreach J ( `find $DIR -name "*.jar"` )
        printf "."
        set res=`jar tvf $J | grep "$CLASS" | sed 's/$/;/'`
        set outputsize = `echo $res|wc -c`
        if ( $outputsize > 1 )  then
                printf "\nMatch found in [$J]\n"
                printf  "$res\n" | tr ';' '\n'
        endif
end

printf "\n"
Categories: Java, Shell Scripting, Tools, UNIX Tags: , , , ,

Changing a service’s startup policy via CLI

April 16, 2012 Leave a comment

I recently needed to automate an installation of a service but ran into a problem that it was starting automatically by default (which we did not want).

So I found a simple CLI to change this behavior –

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\my-service-name /v Start /t REG_DWORD /d 3 /f

What it does is:

  1. Go to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\my-service-name entry in the registry
  2. Sets the value of Start (/v Start) to be the 32bit value (/t REG_DWORD) of 3 (/d 3)
  3. It overrides the current value if exists (/f)
All valid values are:
  • 2 = Automatic
  • 3 = Manual
  • 4 = Disabled
I have to give credit to this Microsoft article: http://support.microsoft.com/kb/248660

Manipulating the values of environment variables in windows batch files

April 3, 2012 Leave a comment

Lets say you have an environment variable that contains a certain value and you want to change it in some manner.
Example:

set AAA=abc_123
set BBB=%AAA:~0,3%%AAA:~4,3%
echo %BBB%

The output will be abc123
The syntax is: %var:~<offset>,<count>%
Examples:
echo %AAA:~0,1% – will print “a” – Start at offset 0 and count 1 character.
echo %AAA:~1,2% – will print “bc” – Start at offset 1 and count 2 characters.
echo %AAA:~3,3% – will print “_12” – Start at offset 3 and count 3 characters.
echo %AAA:~1,-2% – will print “bc_1” – Start at offset 1 and count 2 characters from the end.

Another option is

set AAA=abc_123
set BBB=%AAA:abc=ABC%
echo %BBB%

The output will be ABC_123
The syntax is: %var:<string to find>=<alternate value>%
Examples:
echo %AAA:a=A% – will print “Abc_123”.
echo %AAA:_=-% – will print “abc-123”

There are more options – just run help set on the command line to find them out.

%d bloggers like this: