SED but true

May 15, 2015 Leave a comment

Working on a *nix like system for enough time will get you exposed to the sed command line tool. sed – which stands for “Stream Editor” – is in my opinion one of the most feared command line tools available on *nix environments, mostly since when you see someone using it – it seems like they are just typing a meaningless stream of slashes, back-slashes and quotation marks. However, this is a very powerful tool that can help you manipulate text files and streams relatively easily without much understanding of regular expressions.
The most basic capability in sed is the ability to search and replace values – and the syntax is rather simple: sed 's/value to search for/value to put instead/g'.
The s at the beginning is mandatory (for search and replace) – think of it as saying “Search-and-replace”. The g at the end is optional and stands for “Global” – it tells sed whether to replace all matches in a line or just the first match (in a line). Here are some examples of simple find and replace scenarios.

~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/'
Hello sir, this indeed a confusing example
~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/g'
Hello sir, this indeir a confusing example

Usually, you would need some understanding of regular expressions (regex) to do advanced operations:

~> echo 'Hello sir, this indeed a confusing example' | sed 's/sir.*/world/'
Hello world

In the above example we tell sed to find the text ‘sir’ with zero or more characters after it – and replace these with the text ‘world’. The designation ‘.*’ in regex means zero or more characters of any kind (letters, punctuation, numbers, white spaces etc).

Regex Groups

A useful (yet confusing) ability related to regular expressions is support of groups. Groups in regex essentially mean matches of the regular expression – so if we take the text “A=B”, and apply the regular expression: “(.*)=(.*)”, we will get two groups the first with the value “A” and the second with the value “B”. Unfortunately – sed doesn’t want us to have a simple life – so “(.*)=(.*)” is actually “\(.*\)=\(.*\)”. Once I matched these groups – I can use them in the “value to put instead” part of the sed command – they can be referred to as \x – where x is the index of the group. so \1 is the first group, and so on. Example:

~> echo "A=B" | sed 's/\(.*\)=\(.*\)/If \1=\2, I can assert that \2=\1 as well/'
If A=B, I can assert that B=A as well
Useful sed Patterns

Here are some patterns I found useful:

~> echo "Everything in this line is a secret, apart from this" | sed 's/.*from //'
this
~> echo 'I would like to thank my kids, my mother, my boss, my neighbors and most importantly - my wife!' | sed 's/ and most.*/./'
I would like to thank my kids, my mother, my boss, my neighbors.
~> echo "A1B2C3D4" | sed 's/[0-9]*//g'
ABCD
~> echo "My phone number is: +1-212-555-9876" | sed 's/[^0-9\+\-]*//g'
+1-212-555-9876
Advanced Usages

Lacking a better way to express it – I can say that sed has crazy capabilities. Apart from inserting lines, deleting lines it can even run shell commands as part of its work.

I recently wrote myself and ad-hoc template engine based on sed – and this is the entire code:

cp template target

while read line; do
   KEY=$(echo $line | cut -d"=" -f1)
   VALUE=$(echo $line | cut -d"=" -f2)
   echo replacing '$('${KEY}')' with $VALUE
   sed -i '' "s/\$(${KEY})/${VALUE}/g" target
done < key_value_input_file

You can see a working example of the code here: http://ideone.com/ISxTMj

References

Groovy – Easy XML Parsing with the XmlSlurper

August 4, 2014 Leave a comment

Groovy comes built-in with a cool feature for parsing XML, and easily “objectifying” it.
Its a one-liner:

def document = new XmlSlurper().parseText(xml); // Convert the XML into an Object Model

Usage example

import groovy.util.*;

public class GroovyXmlSlurperExample
{
   public static void main(String[] args)
   {
      def xml = "<root>\n\t<item name=\"a\" value=\"1\"/>\n\t<item name=\"b\" value=\"2\"/>\n</root>";
      System.out.println("XML is: \n" + xml);

      def document = new XmlSlurper().parseText(xml); // Convert the XML into an Object Model

      System.out.println("\nElements are:");
      for (item in document.item)
      {
         System.out.println(item.@name.text() + "=" + item.@value.text());
      }
        
      System.out.println("\nJust the value of the 'value' attributes: " + document."item".@value.text());
   }
}

Explanation

Line #10 – parses the XML from a String to an actual object.
Line #13 – iterates over all the elements of type “item” under the root.
Line #15 – shows how to get the value of attributes for a certain element, the ‘@name’ syntax refers to an attribute named “name” (... name="a" ...)

Further Reading & Experiments

Categories: Groovy, XML Tags: , , , , ,

Maven snippet – add generated sources to the build

July 21, 2014 Leave a comment

Here’s a maven snippet I found myself looking for many times – adding generated sources to the build.
This tells Maven to include the generated classes as part of the compilation, which ensures any non generated code that depends on them compiles fine, and their corresponding .class files be included in the .jar/.war file.

  ...
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>add-source</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>add-source</goal>
            </goals>
            <configuration>
              <sources>
                <source>${project.build.directory}/generated</source>
              </sources>
            </configuration>
          </execution>
        </executions>
      </plugin>
    ...
    </plugins>
  </build>
  ...

In this case (line #153) – we assume that the sources are generated to target/generated.

Some useful reading material regarding this subject:

Bash Epoch Converter

July 10, 2014 Leave a comment

I use the online epoch converter on an almost daily basis with an almost single usage – converting a value of seconds/milliseconds since the epoch (1/1/1970 00:00:00) to a human readable date.
I wanted something quicker which I can use from my Terminal window without – so I created a Bash function that performs that:

epoc usage

epoc usage


The function

epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')"
}
epoc ()
{
    [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
    [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1;
    [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
    [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
    printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
    printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')"
}

One liners:

# UNIX, Mac OSx and FreeBSD
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" ; }
# Linux
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" ; }

Personalizing your connection to QA/Production environments with SecureCRT

January 6, 2014 Leave a comment

The trouble with production/QA environments is that you can rarely change them to fit you personal preferences, and usually you use a common user to access them.
If you are working on Windows and using SecureCRT to connect to Linux/Unix machines (and if not – ask yourself why …) you can personalize any connection to QA/Production environments by telling SecureCRT to perform automatic login activities – this without effecting how the machine behaves for other users.
By personalization I am referring to the definition of your personal aliases etc.

How to do it:

  1. Download this VBScript to your computer, and place it somewhere (e.g. C:\Utils\SecureCRT\Scripts)
  2. Find a connection you want to effect in SecureCRT and open it’s properties
  3. Navigate to Connection -> Logon Actions
  4. Check the Logon script checkbox and type the path to the file defined at the beginning (e.g. C:\Utils\SecureCRT\Scripts\EnvironmentPersonalizer.vbs)

Every time you login – SecureCRT will run that script.

What the script does:
The way this scripts works is that it waits for something to happen on the screen and then sends commands.
Example:

objTab.Screen.WaitForString("$")   // Wait for the prompt
objTab.Screen.Send "bash" & vbcr   // Switch to a BASH shell
objTab.Screen.Send "clear" & vbcr   // clear the screen
objTab.Screen.Send "alias ll='ls -lart'" & vbcr   // Alias 'll' to be 'ls –lart' 

You can find a few more utility scripts on my Github Repository – https://github.com/ronkitay/SecureCRT-Environment-Personalization

Note: Take great care when effecting anything on production environments – don’t do anything that will effect how the environment behaves for other people (like changing the .bash_profile for example)

Categories: Shell Scripting, Tools, UNIX Tags:

Automatically setting permanent Environment Variables on Windows

August 26, 2013 Leave a comment

More than once I needed to define a permanent Environment Variable on a Windows machine – mostly as part of some automatic installation process. E.g. – like the Java installer sets the JAVA_HOME during installation.

Windows has a very simple utility to do so – SETX.

# Set the variable HELLO with the value "WORLD" for the current user
SETX HELLO WORLD
# Set the variable HELLO with the value "WORLD" for all users (System)
SETX HELLO WORLD /m

Note that SETX is different from SET – invoking SETX as part of a shell script will not make the variable available for that shell script.

Reading material
Official Documentation – http://technet.microsoft.com/en-us/library/cc755104.aspx
Non-Official Documentation – http://ss64.com/nt/setx.html

Demeter is running

August 19, 2013 Leave a comment

I think that one of the widely ignored laws in in object oriented programming is the Law of Demeter (LoD).

Take the following real life scenario:

John needs $5. John goes to Jack, pulls out his wallet an opens it – and takes $5, than puts the wallet back in its palace.

Sounds bad – right? Now see it as a piece of code and think if you ever saw something like this in your source code:

public class Wallet
{
     private int amount;

     public Wallet(int amount)
     {
          this.amount = amount;
     }

     public int deposit(int sum)
     { 
         if (sum < 0) throw new IllegalArgumentException("sum must be non-negative");
         amount += sum;
         return amount;
     }
     public int withdraw(int sum)
     { 
         if (sum < 0) throw new IllegalArgumentException("sum must be non-negative");
         if (sum > amount) throw new InsufficiantFundsException("Insufficiant Funds!");
         amount -= sum;
         return amount;
     }
}

public class Person
{
     private string name;
     private Wallet wallet;

     public Person(String name, int moneyInWallet)
     {
          this.name = name;
          this.wallet = new Wallet(moneyInWallet);
     }

     public boolean requestLoanFrom(Person otherPerson, int amount)
     {
          int moneyFromOtherPerson = otherPerson.getWallet().withdraw(amount);
          this.wallet.deposit(moneyFromOtherPerson);          
          
          return true;
     }
}

public class Main
{
     public static void main(String [] args)
     {
          Person john = new Person("John", 10);
          Person jack = new Person("jack", 10);

          jack.requestLoanFrom(john,5);          
     }
}

The LoD relates to the fact that Jack knows John’s wallet – which it should not.

Now consider this implementation:

public class Person
{
     ...

     public boolean requestLoanFrom(Person otherPerson, int amount)
     {
          if (otherPerson.handleLoanRequest(amount))
          {
              this.wallet.deposit(amount);          
              return true;
          }
          else
          {return false;}                  
     }

     protected boolean handleLoanRequest(int amount)
     {
          // Do some validation ...
          wallet.withdraw(amount);     
          return true;
     }
}

Notice that each person is responsible for their own wallet – as it is in real life*.
*Note: I’m well aware this code is bad from the transaction POV – it is just to clarify the point.

To simplify things, the LoD states that this line of code is bad:

x = a.getSome().getThing();

It basically states that your code should be like:

public class A
{
   private Some some;

   public Thing getSomeThing()
   {
       this.some.getThing();
   }
}

public class Main
{
   public static void main(String ... args)
   {
      x = a.getSomeThing();
   }
}

I believe LoD is a useful guideline when developing Object Oriented software, but becoming too strict with it has a price.
In my experience – when you find it very hard to adhere to the LoD – you should review your design as you might be tight-coupling your system.

Follow

Get every new post delivered to your Inbox.

Join 236 other followers

%d bloggers like this: