SED but true
Working on a *nix like system for enough time will get you exposed to the sed command line tool. sed – which stands for “Stream Editor” – is in my opinion one of the most feared command line tools available on *nix environments, mostly since when you see someone using it – it seems like they are just typing a meaningless stream of slashes, back-slashes and quotation marks. However, this is a very powerful tool that can help you manipulate text files and streams relatively easily without much understanding of regular expressions.
The most basic capability in sed is the ability to search and replace values – and the syntax is rather simple: sed 's/value to search for/value to put instead/g'.
The s at the beginning is mandatory (for search and replace) – think of it as saying “Search-and-replace”. The g at the end is optional and stands for “Global” – it tells sed whether to replace all matches in a line or just the first match (in a line). Here are some examples of simple find and replace scenarios.
~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/' Hello sir, this indeed a confusing example
~> echo 'Hello sed, this indeed a confusing example' | sed 's/ed/ir/g' Hello sir, this indeir a confusing example
Usually, you would need some understanding of regular expressions (regex) to do advanced operations:
~> echo 'Hello sir, this indeed a confusing example' | sed 's/sir.*/world/' Hello world
In the above example we tell sed to find the text ‘sir’ with zero or more characters after it – and replace these with the text ‘world’. The designation ‘.*’ in regex means zero or more characters of any kind (letters, punctuation, numbers, white spaces etc).
Regex Groups
A useful (yet confusing) ability related to regular expressions is support of groups. Groups in regex essentially mean matches of the regular expression – so if we take the text “A=B”, and apply the regular expression: “(.*)=(.*)”, we will get two groups the first with the value “A” and the second with the value “B”. Unfortunately – sed doesn’t want us to have a simple life – so “(.*)=(.*)” is actually “\(.*\)=\(.*\)”. Once I matched these groups – I can use them in the “value to put instead” part of the sed command – they can be referred to as \x – where x is the index of the group. so \1 is the first group, and so on. Example:
~> echo "A=B" | sed 's/\(.*\)=\(.*\)/If \1=\2, I can assert that \2=\1 as well/' If A=B, I can assert that B=A as well
Useful sed Patterns
Here are some patterns I found useful:
~> echo "Everything in this line is a secret, apart from this" | sed 's/.*from //' this
~> echo 'I would like to thank my kids, my mother, my boss, my neighbors and most importantly - my wife!' | sed 's/ and most.*/./' I would like to thank my kids, my mother, my boss, my neighbors.
~> echo "A1B2C3D4" | sed 's/[0-9]*//g' ABCD
~> echo "My phone number is: +1-212-555-9876" | sed 's/[^0-9\+\-]*//g' +1-212-555-9876
Advanced Usages
Lacking a better way to express it – I can say that sed has crazy capabilities. Apart from inserting lines, deleting lines it can even run shell commands as part of its work.
I recently wrote myself and ad-hoc template engine based on sed – and this is the entire code:
cp template target
while read line; do
KEY=$(echo $line | cut -d"=" -f1)
VALUE=$(echo $line | cut -d"=" -f2)
echo replacing '$('${KEY}')' with $VALUE
sed -i '' "s/\$(${KEY})/${VALUE}/g" target
done < key_value_input_file
You can see a working example of the code here: http://ideone.com/ISxTMj
References
- Online Regex Parser + Tutor: http://www.regexr.com/
- Regex Debugger + Visualizer: https://www.debuggex.com/
- Regex Reference page – http://www.regular-expressions.info/tutorialcnt.html
- sed man (Manual) page – http://ss64.com/bash/sed.html
- sed useful one-liners: http://sed.sourceforge.net/sed1line.txt
Groovy – Easy XML Parsing with the XmlSlurper
Groovy comes built-in with a cool feature for parsing XML, and easily “objectifying” it.
Its a one-liner:
def document = new XmlSlurper().parseText(xml); // Convert the XML into an Object Model
Usage example
import groovy.util.*;
public class GroovyXmlSlurperExample
{
public static void main(String[] args)
{
def xml = "<root>\n\t<item name=\"a\" value=\"1\"/>\n\t<item name=\"b\" value=\"2\"/>\n</root>";
System.out.println("XML is: \n" + xml);
def document = new XmlSlurper().parseText(xml); // Convert the XML into an Object Model
System.out.println("\nElements are:");
for (item in document.item)
{
System.out.println(item.@name.text() + "=" + item.@value.text());
}
System.out.println("\nJust the value of the 'value' attributes: " + document."item".@value.text());
}
}
Explanation
Line #10 – parses the XML from a String to an actual object.
Line #13 – iterates over all the elements of type “item” under the root.
Line #15 – shows how to get the value of attributes for a certain element, the ‘@name’ syntax refers to an attribute named “name” (... name="a" ...)
Further Reading & Experiments
- Experiment online – just fork this code and play around: https://ideone.com/g2C7Tx
- Groovy’s tutorial for XML Slurper
- XmlSlurper “GroovyDoc” page
Maven snippet – add generated sources to the build
Here’s a maven snippet I found myself looking for many times – adding generated sources to the build.
This tells Maven to include the generated classes as part of the compilation, which ensures any non generated code that depends on them compiles fine, and their corresponding .class files be included in the .jar/.war file.
...
<build>
<plugins>
...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${project.build.directory}/generated</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
...
In this case (line #153) – we assume that the sources are generated to target/generated.
Some useful reading material regarding this subject:
Bash Epoch Converter
I use the online epoch converter on an almost daily basis with an almost single usage – converting a value of seconds/milliseconds since the epoch (1/1/1970 00:00:00) to a human readable date.
I wanted something quicker which I can use from my Terminal window without – so I created a Bash function that performs that:
epoc usage
The function
epoc ()
{
[[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
[[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1;
[[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
[[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')"
}
epoc ()
{
[[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s);
[[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1;
[[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1;
[[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000;
printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)";
printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')"
}
One liners:
# UNIX, Mac OSx and FreeBSD
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold)epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" "$(export TZ=GMT; date -r ${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%z)' | sed 's/(+\([0-9][0-9]\)/(+\1:/')" ; }
# Linux
function epoc() { [[ $# == 0 ]] && SECONDS_SINCE_EPOCH=$(date +%s); [[ "$1" == "-h" ]] && printf "$(tput bold)$(tput setaf 1)Usage:$(tput sgr0) $(tput bold) epoc [<seconds-since-epoch|milli-seconds-since-epoch>]$(tput sgr0)\n$(tput bold)$(tput setaf 2)Examples:$(tput sgr0)\n\t$(tput bold)epoc 1404382131$(tput sgr0)\n\t$(tput bold) epoc 1404305405000\n" && return 1; [[ $# == 1 ]] && SECONDS_SINCE_EPOCH=$1; [[ $(printf ${SECONDS_SINCE_EPOCH} | wc -c) -gt 10 ]] && let SECONDS_SINCE_EPOCH=${SECONDS_SINCE_EPOCH}/1000; printf "\n$(tput bold)$(tput bold)$(tput setaf 2)%-20s\t| %-35s\t| %-35s$(tput sgr0)\n" "Seconds Since Epoch" "Time (Local)" "Time (GMT)"; printf "%-20s\t| %-35s\t| %-35s\n\n" ${SECONDS_SINCE_EPOCH} "$(date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" "$(export TZ=GMT; date -d @${SECONDS_SINCE_EPOCH} '+%d-%h-%Y %H:%M:%S %Z (%:z)')" ; }
Personalizing your connection to QA/Production environments with SecureCRT
The trouble with production/QA environments is that you can rarely change them to fit you personal preferences, and usually you use a common user to access them.
If you are working on Windows and using SecureCRT to connect to Linux/Unix machines (and if not – ask yourself why …) you can personalize any connection to QA/Production environments by telling SecureCRT to perform automatic login activities – this without effecting how the machine behaves for other users.
By personalization I am referring to the definition of your personal aliases etc.
How to do it:
- Download this VBScript to your computer, and place it somewhere (e.g. C:\Utils\SecureCRT\Scripts)
- Find a connection you want to effect in SecureCRT and open it’s properties
- Navigate to Connection -> Logon Actions
- Check the Logon script checkbox and type the path to the file defined at the beginning (e.g. C:\Utils\SecureCRT\Scripts\EnvironmentPersonalizer.vbs)
Every time you login – SecureCRT will run that script.
What the script does:
The way this scripts works is that it waits for something to happen on the screen and then sends commands.
Example:
objTab.Screen.WaitForString("$") // Wait for the prompt
objTab.Screen.Send "bash" & vbcr // Switch to a BASH shell
objTab.Screen.Send "clear" & vbcr // clear the screen
objTab.Screen.Send "alias ll='ls -lart'" & vbcr // Alias 'll' to be 'ls –lart'
You can find a few more utility scripts on my Github Repository – https://github.com/ronkitay/SecureCRT-Environment-Personalization
Note: Take great care when effecting anything on production environments – don’t do anything that will effect how the environment behaves for other people (like changing the .bash_profile for example)
Automatically setting permanent Environment Variables on Windows
More than once I needed to define a permanent Environment Variable on a Windows machine – mostly as part of some automatic installation process. E.g. – like the Java installer sets the JAVA_HOME during installation.
Windows has a very simple utility to do so – SETX.
# Set the variable HELLO with the value "WORLD" for the current user SETX HELLO WORLD # Set the variable HELLO with the value "WORLD" for all users (System) SETX HELLO WORLD /m
Note that SETX is different from SET – invoking SETX as part of a shell script will not make the variable available for that shell script.
Reading material
Official Documentation – http://technet.microsoft.com/en-us/library/cc755104.aspx
Non-Official Documentation – http://ss64.com/nt/setx.html
Demeter is running
I think that one of the widely ignored laws in in object oriented programming is the Law of Demeter (LoD).
Take the following real life scenario:
John needs $5. John goes to Jack, pulls out his wallet an opens it – and takes $5, than puts the wallet back in its palace.
Sounds bad – right? Now see it as a piece of code and think if you ever saw something like this in your source code:
public class Wallet
{
private int amount;
public Wallet(int amount)
{
this.amount = amount;
}
public int deposit(int sum)
{
if (sum < 0) throw new IllegalArgumentException("sum must be non-negative");
amount += sum;
return amount;
}
public int withdraw(int sum)
{
if (sum < 0) throw new IllegalArgumentException("sum must be non-negative");
if (sum > amount) throw new InsufficiantFundsException("Insufficiant Funds!");
amount -= sum;
return amount;
}
}
public class Person
{
private string name;
private Wallet wallet;
public Person(String name, int moneyInWallet)
{
this.name = name;
this.wallet = new Wallet(moneyInWallet);
}
public boolean requestLoanFrom(Person otherPerson, int amount)
{
int moneyFromOtherPerson = otherPerson.getWallet().withdraw(amount);
this.wallet.deposit(moneyFromOtherPerson);
return true;
}
}
public class Main
{
public static void main(String [] args)
{
Person john = new Person("John", 10);
Person jack = new Person("jack", 10);
jack.requestLoanFrom(john,5);
}
}
The LoD relates to the fact that Jack knows John’s wallet – which it should not.
Now consider this implementation:
public class Person
{
...
public boolean requestLoanFrom(Person otherPerson, int amount)
{
if (otherPerson.handleLoanRequest(amount))
{
this.wallet.deposit(amount);
return true;
}
else
{return false;}
}
protected boolean handleLoanRequest(int amount)
{
// Do some validation ...
wallet.withdraw(amount);
return true;
}
}
Notice that each person is responsible for their own wallet – as it is in real life*.
*Note: I’m well aware this code is bad from the transaction POV – it is just to clarify the point.
To simplify things, the LoD states that this line of code is bad:
x = a.getSome().getThing();
It basically states that your code should be like:
public class A
{
private Some some;
public Thing getSomeThing()
{
this.some.getThing();
}
}
public class Main
{
public static void main(String ... args)
{
x = a.getSomeThing();
}
}
I believe LoD is a useful guideline when developing Object Oriented software, but becoming too strict with it has a price.
In my experience – when you find it very hard to adhere to the LoD – you should review your design as you might be tight-coupling your system.
Chrome’s Search Engines
A good way to increase effectivity when browsing is to use Google Chrome’s “Search Engines” functionality.
This gives you a way to quickly search within sites that have search options, or to navigate to to sites where you know their structure.
How to do it? three steps:
- Click on the ‘Settings’ button.
- Choose ‘Manage Search Engines’
- Scroll down until you see editable text boxes and give 3 values for your search engine:
- The name of the search engine – e.g. “Wikipedia”
- The alias for this search engine – e.g. “wiki”
- The search string for this search engine – e.g. “http://en.wikipedia.org/w/index.php?search=%s&button=&title=Special%3ASearch”
Just place ‘%s’ where the site you use expects the term to search for.
Example from life: Let’s say you work a lot on a Unix/Linux environment and need to man a lot of commands. You can view these man pages online with much better detail and examples on this site:
http://ss64.com/bash/.
Every page on this site has the following pattern: http://ss64.com/bash/command.html
All you need to do is define a new search engine like this:
- Name: UNIX Man Pages
- Alias: man
- URL: http://ss64.com/bash/%s.html
That way when you type “man less” – you will see this: 
Pressing enter will lead you to the requested page:
Teaching Ditto to paste into SecureCRT
I use Ditto and SecureCRT constantly on my day to day work, but by default – they don’t work well together.
Ditto supports being configured for specific applications to work in a different manner – all you need to do is to update the registry in the correct way.
How to do it?
Automatically
- Save the below into a file named
Ditto_SecureCRT.reg - Execute the file and choose Yes when prompted
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Ditto\PasteStrings]
"SecureCRT.exe"="+{INS}"
Manually
- Open the registry editor (Start -> Run -> regedit)
- Navigate to
HKEY_CURRENT_USER\Software\Ditto - If the Key PasteStrings does not exist, create it: Right-Click on Ditto -> New -> Key -> Name it as ‘PasteStrings’
- Add a new value for SecureCRT:
- Right-Click on PasteStrings -> New -> String Value -> Name it as ‘SecureCRT.exe’
- Double click on the new value, and give Value Data the value
+{INS}
Either way you do it – the change is immediate, no need for a restart.
Dev (Tool)Tips 