Encoding information with barcodes

Posted felixPosted in laboratory, technology

Barcodes are not just for supermarkets! Any medium to high-throughput laboratory can benefit greatly from employing barcodes. The process of identifying and tracking samples and all types of other things (please don’t barcode your technician!) through locations and processes  can:

  • Save time typing the entries manually
  • Greatly reduce the risks of mis-identification
  • Allow longer IDs (i.e. more samples) or more other data without more effort

Traditional barcodes exist as one-dimensional and two-dimensional black-and-white images (some examples in fig. 1). Both types are read using an optical sensor at a set distance. This could be a dedicated barcode-scanner or the camera of your smartphone. One-Dimensional barcodes typically track just one piece of data, such as the sample number. Two-Dimensional barcodes are able to encode more data, including sample ID, customer ID, lot numbers, and more.

There are around 100 different encoding schemes for barcodes. The original standard are “EAN/UPC” linear barcodes, but “Code-128” is becoming more established as it allows letters as well as digits (all 128 ASCII code characters), includes a start and stop identification. The latter as well as a “quite zone” (white space) around the barcode and a checksum digit at the end of the code make the reading more reliable, reducing the risk of mis-identification.
An online resource to see all the different types of barcodes and try encoding information with them can be found here, a specification document is also provided. A whole lecture about the science of barcodes is provided here. Two examples of encoding “Gene-Test” can be seen in fig. 2 & 3:

2-D Codablock-F encoding
Linear EAN128 barcode

In the EAN-128 scheme there are four types of bar sizes, one character is represented by 3 bars and 3 spaces (total six elements). Using the table here we can decode this barcode:

 

Sources and further information:
www.csolsinc.com, barcode.tec-it.com

 

 

Using Container Software

Posted felixPosted in bioinformatics, container, software, technology, unix

One of the amazingly useful current trends in software development is “containerization“. This describes setting up a self-contained environment on a host computer system, that can run a separate operating system (OS), contain data and software not usually available otherwise. I find this very appealing e.g. for

  • testing a software package that is not available for my usual OS
  • testing my own software on a different OS
  • developing or running an analysis in fully reproducible settings
  • sharing software or analysis with reproducible settings

The main players in the field are Docker and Singularity. There are pros and cons for each, Singularity might be better suited for shared environments as you can run the containers with standard user rights.
The general idea of containerization is similar to virtual machines, this Docker page explains the differences.

Docker

To install Docker on an Ubuntu system, currently the following commands will do the trick:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" 
sudo apt-get update
sudo apt-get install -y docker-ce

Installation of Docker on Mac OSX is straightforward with the download of the free community edition (CE) from here. Make sure you either add the software as a log-in item or manually start it before you attempt to use it. (You will see the little Docker whale in the task bar. )

Some key Docker commands are

# show all images available locally:
docker images
# show all containers that are already running:
docker ps
# start a new container from an image that can be fetched from the remote Docker repository.
# a simple test:
docker run hello-world
# or a small Linux system:
docker run --rm -it phusion/baseimage:0.9.22 /sbin/my_init -- bash

A good way to move data in and out of the container is by mounting a specific directory, e.g./home/felix/testproject to the tmp folder:

docker run --rm -it --mount type=bind,source=/home/felix/testproject,target=/tmp phusion/baseimage:0.9.22 /sbin/my_init -- bash

The standard way to create new images is by defining all installation steps in a Dockerfile. However, to share a pre-build environment with your own data it is sometimes easier to freeze the container you have build up. Your can do this by starting a base system, e.g. the phusion/baseimage mentioned above, installing all the software and data you like and exporting the container:

# find container ID:
docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED       
c24c11050fc5        phusion/baseimage:0.9.22   "/sbin/my_init -- ..."   4 seconds ago 
# export the current state in a compressed archive:
docker export c24c11050fc5 | gzip > my_new_image.tar.gz
# import and run the image later or on a different computer:
docker import my_new_image.tar.gz my_container
docker run -it my_container /sbin/my_init -- bash

 

Singularity

Installation on Mac OSX requires the Vagrant and the VirtualBox system first. Instead of using the brew system often recommended, I found it better to install the dmg packages directly from the Vagrant site and the Oracle VirtualBox site. Good instructions for different systems are also given here.
After installation you can start the VirtualBox Manager and set up an Ubuntu image:

# get the image:
vagrant init ubuntu/trusty64
# start the virtual machine:
vagrant up
# get to the command line:
vagrant ssh

or directly start a Singularity image with:

vagrant init singularityware/singularity-2.4
vagrant up
vagrant ssh

You can stop a VM running in the background with

vagrant halt

It is possible to use Docker images in Singularity, you can pull from the Docker hub and build a local image:

sudo singularity build phusion-singularity-image.simg docker://phusion/baseimage:0.9.22

 

This should get you started with containers.
Make sure to keep an eye on disk consumption, in particular Docker data seems to grow significantly in the background (See issue here)! I prefer to move Docker (the “qcow2” file) to a fast external disk.

Checking files & directories in your Bash script

Posted felixPosted in unix

It’s good practise to perform all kinds of tests on files that you are processing in a Bash script, in particular if it is a file that the user has provided. Here is a reminder of the test flags you can use as part of the test command:

Switches to check:

-b filename – Block special file
-c filename – Special character file
-d directoryname – Check for directory Existence
-e filename – Check for file existence, regardless of type (node, directory, socket, etc.)
-f filename – Check for regular file existence not a directory
-G filename – Check if file exists and is owned by effective group ID
-G filename set-group-id – True if file exists and is set-group-id
-k filename – Sticky bit
-L filename – Symbolic link
-O filename – True if file exists and is owned by the effective user id
-r filename – Check if file is a readable
-S filename – Check if file is socket
-s filename – Check if file is nonzero size
-u filename – Check if file set-user-id bit is set
-w filename – Check if file is writable
-x filename – Check if file is executable

How to use:

#!/bin/bash
file=./file
if [ -e "$file" ]; then
    echo "File exists"
else 
    echo "File does not exist"
fi

A test expression can be negated by using the ! operator

#!/bin/bash
file=./file
if [ ! -e "$file" ]; then
    echo "File does not exist"
else 
    echo "File exists"
fi

Source: GroundZero @ StackOverflow

Additionally it might be useful here to learn additional comparison operators:

[ ( EXPR ) ] Returns the value of EXPR. This may be used to override the normal precedence of operators.
[ EXPR1 -a EXPR2 ] True if both EXPR1 and EXPR2 are true.
[ EXPR1 -o EXPR2 ] True if either EXPR1 or EXPR2 is true.
[ -z STRING ] True of the length if “STRING” is zero.
[ -n STRING ] or [ STRING ] True if the length of “STRING” is non-zero.
[ STRING1 == STRING2 ] True if the strings are equal. “=” may be used instead of “==” for strict POSIX compliance.
[ STRING1 != STRING2 ] True if the strings are not equal.
[ ARG1 OP ARG2 ] “OP” is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if “ARG1” is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to “ARG2”, respectively. “ARG1” and “ARG2” are integers.

Source

Allow another user to connect to my EC2 cloud machine

Posted felixPosted in cloud computing, unix

There might be times when you either have something fascinating happening on the Amazon cloud machine that you set up, or – more likely – you got stuck with a problem a friend might be able to help you with. Here are notes on how to set up a shared access for a standard (Ubuntu linux) computer.

If you set up your EC2 machine securely, it will not allow anyone but you to access it though: The “security group” used allows your IP only.  (If you set it up insecurely with Source = 0.0.0.0/0 your friend – and anyone else – will be able to access it directly!) There is however the option to modify this security group, and after a few minutes delay it will be applied even to running machines! So all you need to do to allow your friend to work alongside of you is to add his IP address and add his public SSH key:

Part 1 (see screenshot below):

  • Find out the additional IP address, e.g. with www.iplocation.net/find-ip-address from your friend’s computer, e.g. 79.217.24.86
  • Go to the EC2 management console
  • In the navigation pane, choose Network Interfaces.
  • Select the network interface and choose Actions >> Edit inbound rules
  • Add a new SSH rule with the additional IP address, e.g. 79.217.24.86/32 and save it. 

Part 2:

  • copy his public SSH key to the machine (using your own private key):

    scp -i "~/.ssh/your-key-region1.pem" friends_key.pub ubuntu@ec2-54-236-163-221.compute-1.amazonaws.com:~/
  • log into the machine (with your own private key):

    ssh -i "~/.ssh/your-key.pem" ubuntu@ec2-54-236-163-221.compute-1.amazonaws.com
  • add the new key to your existing keys:

    cat friends_key.pub >> .ssh/authorized_keys

He can now log in with his own (private) key:

ssh -i "~/.ssh/friends_key.pem" ubuntu@ec2-54-236-163-221.compute-1.amazonaws.com

 

This description was based on this help page.

RNA-Seq data quality scores

Posted felixPosted in bioinformatics, sequencing

There are different ways to encode the quality scores in FASTQ files from Next-generation sequencing machines. It is important to find out before using the data and to convert between formats if necessary.

  • Sanger format can encode a Phred quality score from 0 to 93 using ASCII characters 33 to 126 (although in raw read data the Phred quality score rarely exceeds 60, higher scores are possible in assemblies or read maps).
  • Illumina 1.3+ format can encode a Phred quality score from 0 to 62 using ASCII characters 64 to 126 (although in raw read data Phred scores from 0 to 40 only are expected).
  • Solexa/Illumina 1.0 format can encode a Solexa/Illumina quality score from -5 to 62 using ASCII characters 59 to 126 (although in raw read data Solexa scores from -5 to 40 only are expected)
  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.....................................................
  ..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................
  ...............................IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII......................
  .................................JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ......................
  LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL....................................................
  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
  |                         |    |        |                              |                     |
 33                        59   64       73                            104                   126


 S - Sanger        Phred+33,  raw reads typically (0, 40)
 X - Solexa        Solexa+64, raw reads typically (-5, 40)
 I - Illumina 1.3+ Phred+64,  raw reads typically (0, 40)
 J - Illumina 1.5+ Phred+64,  raw reads typically (3, 40)
    with 0=unused, 1=unused, 2=Read Segment Quality Control Indicator
 L - Illumina 1.8+ Phred+33,  raw reads typically (0, 41)

Source: wikipedia

For a simple look-up from ASCII to numeric scores you can use the following list:

ASCII	numeric		ASCII	numeric
!	0		@	31
"	1		A	32
#	2		B	33
$	3		C	34
%	4		D	35
&	5		E	36
'	6		F	37
(	7		G	38
)	8		H	39
*	9		I	40
+	10		J	41
,	11		K	42
-	12		L	43
.	13		M	44
/	14		N	45
0	15		O	46
1	16		P	47
2	17		Q	48
3	18		R	49
4	19		S	50
5	20		T	51
6	21		U	52
7	22		V	53
8	23		W	54
9	24		X	55
:	25		Y	56
;	26		Z	57
<	27		[	58
=	28		\	59
>	29		]	60
?	30		^	61

You can convert the Solexa read quality to Sanger read quality with Maq:

maq sol2sanger s_1_sequence.txt s_1_sequence.fastq

where s_1_sequence.txt is the Solexa read sequence file. Missing this step will lead to unreliable SNP calling when aligning reads with Maq.

Source: maq-manual

Phred itself is a base calling program for DNA sequence traces developed during the initial automation phase of the sequencing of the human genome.
After calling bases, Phred examines the peaks around each base call to assign a quality score to each base call. Quality scores range from 4 to about 60, with higher values corresponding to higher quality. The quality scores are logarithmically linked to error probabilities, as shown in the following table:

Phred quality	Probability of		Accuracy of
score		wrong base call		base call
10 		1 in 10 		90%
20 		1 in 100 		99%
30 		1 in 1,000 		99.9%
40 		1 in 10,000 		99.99%
50 		1 in 100,000 		99.999%

“High quality bases” are usually scores of 20 and above (“Phred20 score”).

You can read the original publications about the Phred program and scoring by Brent Ewing et al. from Phil Green’s lab here and here.
Source: www.phrap.com