Code Snippets

This page contains a small collection of (mainly) bash commands that I discovered during the Ph.D. Bash is evil but during the Ph.D. was often useful to get stuff done quickly (and it is still useful today).

[bash] print a beer icon in a script

    echo $'\360\237\215\272'


    echo $'\360\237\215\270'
    echo $'\360\237\215\271'
    echo $'\360\237\215\273'
    echo $'\360\237\215\274'


[bash] copy the output of a command to the clipboard (on macosx)

    echo "pippo" | pbcopy

Now cmd+v will paste pippo.

[bash] zcat does not work on MACOS (.Z bug)

	sudo mv /usr/bin/zcat /usr/bin/broken-zcat
	sudo ln -s /usr/bin/gzcat /usr/bin/zcat

[bash] awk insert blank line every n lines

    awk '{ if ((NR % 5) == 0) printf("\n"); print; }'

For n == 5, of course. Substitute whatever your idea of n is.

[bash] convert windows source file to unix

	dos2unix *.py

[bash] remove first column from a file

	cut -f2- filename

[bash] replace delete with backspace in screen

You can define an alias in your ~/.bashrc file like:

  alias screen='TERM=screen screen'

[bash] sort using tab as separator

  sort -t$'\t' -k 2 file

[java] change java heap size

  java -Xmx<size> set maximum Java heap size

e.g., java -Xmx4196m Pippo

[bash] resolve redirected url

  REDIRECT_URL=$(curl -w %{redirect_url} URL) 

[bash] get a txt file edited from pirate pad


[bash] dump a website

wget -r -H -l1 -k -P $targetdir --exclude-domains ${comma-seperated domain name} --user=xxx --password=xxx $url

[bash] get a webpage with autentication

url= ""
outpage = ""
curl -u $user:$pass $url >  $outpage

[linux] 10 nice hints

[bash] how to get the sum of all the lines in the file

awk '{a+=$0}END{print a}' abc

[python] remove the garbage collector in python

When you are working with a big data structure in python the garbage collector can slow down the execution, sometime can make sense to disable it.

import gc

[bash] how to query solr from the shell

… Or just use python and the package pysolr.

[bash] sum rows with the some field

dictionary in awk:

$ cat nayan.out
saman 1
gihan 2
saman 4
ravi 1
ravi 2

$ awk '{arr[$1]+=$2} END {for (i in arr) {print i,arr[i]}}' nayan.out > nayan.out.tmp

$ cat nayan.out.tmp
ravi 3
saman 5
gihan 2

[maven] resources in a jar

[bash] get all pdf in a page

  wget -m --accept=pdf -nd  $url

[bash] iterate over the lines of a jar

while read line
echo $line
done < file_to_read

[bash] remove blank lines in a file

grep -v '^[[:space:]]*$'  file

[bash] change tmp folder where sort put temporary files

sort -T dir ...

[bash] print number of a specific char for each line of a file

awk -F'#' '{ print NF-1}' 

It will print how many chars ‘#’ per line

[bash] replace a substring

echo ${i/txt/tex}

It will print pippo.tex

[bash] sort on more than one field

sort -k 1,1 -k 3,3nr file.tsv > tmp

It sorts on the first field and if two elements are equal on the first, on the third field in reverse integer order

[bash] remove all files of size 0

find . -type f -size 0k -exec rm {} \; | awk '{ print $8 }'

[bash] remove all chars in a particular position from a file

cut -c42- file

It removes the first 42 characters from all the lines of a file.

[bash] view what is appended on file

tail -f file -n 

[bash] get words distribution from a file

cat file | tr '[A-Z]' '[a-z]' | tr -sc '[a-z]' '\n' | grep -v '^[^a-z]*$' | grep -v '^[\]'  |    sort | grep -vxf topwords-ita.txt | uniq -c   | sort -nrk1

[bash] tr complement

tr -sc '[a-zA-Z]' '\n' < file

It replaces all chars that are not alphabetic in new lines

[bash] remove first or last lines

 sed '1d' filename 

To remove the first line

 sed '$d' filename 

To remove the last line

[bash] Generate all the couples of terms

tr -sc '[a-zA-Z0-9\n]' '\t'  < inputFile | awk '{for (i = 1; i < NF; i++) print $i"\t"$(i+1); }'

[bash] Find a process listening on a particular port

sudo netstat -lpn | grep :8080

[bash] Check for proper number of command line args


if [ $# -ne $EXPECTED_ARGS ]
  echo "Usage: `basename $0` {arg}"
  exit $E_BADARGS

[bash] Sort in numeric order with scientific notation allowed

sort -g

[bash] create ssh pub key

<ssh-keygen -t rsa -C ""
# Creates a new ssh key, using the provided email as a label
# Generating public/private rsa key pair.
# Enter file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter]

Now you need to enter a passphrase.

Enter passphrase (empty for no passphrase): [Type a passphrase]
# Enter same passphrase again: [Type passphrase again]

Copy in .ssh/authorized_key to login on a remote machine without password.

[bash] gzip a folder

  tar -czf folder_name.tar.gz folder_name/

[git] change the commit message

  git commit --amend -m "New commit message"

[git] undo latest commit (e.g., if you commit -a -m instead of commit -m)

  git reset --soft HEAD^