https://divingintogeneticsandgenomics.com/tags/ngs/ Recent content in NGS on Chatomics Hugo -- gohugo.io en-us © 2018 Ming 'Tommy' Tang Sun, 31 Dec 2017 19:58:00 -0800 https://divingintogeneticsandgenomics.com/archive/the-end-of-2017/ Sun, 31 Dec 2017 19:58:00 -0800 https://divingintogeneticsandgenomics.com/archive/the-end-of-2017/ In the end of last year, I wrote a post summarizing the passing 2016. Now, it is time to write the same for 2017! How time flies! Last year, I wrote: For the coming 2017, I should be: 1. busy with Phoebe. 2. writing 1-2 papers. 3. writing a book chapter on ChIP-seq for the biostar handbook. It should come out in the mid of 2017. 3. writing a small R package for practice. https://divingintogeneticsandgenomics.com/archive/linux-commands-basics/ Wed, 06 Aug 2014 18:18:00 -0700 https://divingintogeneticsandgenomics.com/archive/linux-commands-basics/ I am attending the NGS course at MSU. This is a great course with great instructors and friendly colleagues. I highly recommend this course to everyone. http://bioinformatics.msu.edu/ngs-summer-course-2014 This morning, we learned SNP calling by samtools and sam file specification (I will write another blog for the SNP calling) .in the night , TA Elijah gave an awesome introduction to linux commands. personally, I think this should be taught in the first day of the course. https://divingintogeneticsandgenomics.com/archive/several-ngs-bioinformatics-training-materials/ Wed, 12 Mar 2014 08:21:00 -0700 https://divingintogeneticsandgenomics.com/archive/several-ngs-bioinformatics-training-materials/ 1. Next Generation sequencing wiki book http://en.wikibooks.org/wiki/Next_Generation_Sequencing_(NGS) UCDavis training course https://training.bioinformatics.ucdavis.edu/documentation/ MSU training course by Titus Brown http://ged.msu.edu/angus/index.html GOBLET http://www.mygoblet.org/training-portal  more http://ged.msu.edu/angus/bioinformatics-courses.html a list maintained by Stephen Turner in UVA http://stephenturner.us/edu.html GVL practial protocols for ChIP-seq, RNA-seq, variant calling etc https://genome.edu.au/wiki/GVL These will keep me busy for a while...I feel I do not have enough time to learn :) https://divingintogeneticsandgenomics.com/archive/chip-seq-analysis-programs/ Mon, 20 Jan 2014 12:49:00 -0800 https://divingintogeneticsandgenomics.com/archive/chip-seq-analysis-programs/ This is the first blog in the year of 2014. I am graduating in the coming August, there are so many things to do and I do not have much time to blog. However, I do not want it to be inactive. I have extensive experience with ChIP-seq data analysis, and I want to list all the programs that I usually use. With raw Fastq data, one needs to align it to the reference genome by an aligner. https://divingintogeneticsandgenomics.com/archive/chip-exo-data-analysis/ Sat, 16 Nov 2013 07:58:00 -0800 https://divingintogeneticsandgenomics.com/archive/chip-exo-data-analysis/ Our neighboring lab just generated some ChIP-exo data, (if you do not know the technique, look at this paper http://www.ncbi.nlm.nih.gov/pubmed/23026909) and the company did some analysis but not what they want. The bedgraph files generated by the company were separated for plus strand and minus strand. They want just peak files like in the ChIP-seq experiment. I was aware of several software can be used to deal with this type of data: https://divingintogeneticsandgenomics.com/archive/two-pipelines-for-ngs-analysis/ Mon, 12 Aug 2013 06:11:00 -0700 https://divingintogeneticsandgenomics.com/archive/two-pipelines-for-ngs-analysis/ bcbio-nextgen from Brad Chapman https://github.com/chapmanb/bcbio-nextgen Documentation https://bcbio-nextgen.readthedocs.org/en/latest/index.html Metaseq from Ryan Dale https://github.com/daler/metaseq Documentation http://pythonhosted.org/metaseq/ https://divingintogeneticsandgenomics.com/archive/deseq2-is-now-in-bioconductor/ Thu, 25 Jul 2013 13:26:00 -0700 https://divingintogeneticsandgenomics.com/archive/deseq2-is-now-in-bioconductor/ http://seqanswers.com/forums/showthread.php?t=28350 http://bioconductor.org/packages/2.12/bioc/html/DESeq2.html Thank you Simon and your team! https://divingintogeneticsandgenomics.com/archive/install-bioawk-in-unbuntu/ Sat, 20 Jul 2013 20:21:00 -0700 https://divingintogeneticsandgenomics.com/archive/install-bioawk-in-unbuntu/ Bioawk is written by Heng Li, and it can handle formats like: bed: 1:chrom 2:start 3:end 4:name 5:score 6:strand 7:thickstart 8:thickend 9:rgb 10:blockcount 11:blocksizes 12:blockstarts sam: 1:qname 2:flag 3:rname 4:pos 5:mapq 6:cigar 7:rnext 8:pnext 9:tlen 10:seq 11:qual vcf: 1:chrom 2:pos 3:id 4:ref 5:alt 6:qual 7:filter 8:info gff: 1:seqname 2:source 3:feature 4:start 5:end 6:score 7:filter 8:strand 9:group 10:attribute fastx: 1:name 2:seq 3:qual 4:comment  I followed the tutorial here https://github.com/vsbuffalo/bioawk-tutorial https://divingintogeneticsandgenomics.com/archive/a-python-script-for-extracting-fastq-sequences/ Thu, 04 Jul 2013 09:05:00 -0700 https://divingintogeneticsandgenomics.com/archive/a-python-script-for-extracting-fastq-sequences/ a simple python script to handle fastq file. fast fastq file parser, extract ~10,000 reads with their sequences and quality from a fastq file (~70milion reads) based on names of the reads. HengLi in Princeton https://github.com/lh3/seqtk has a wrapper for this kind of task. the following code is from http://www.biostars.org/p/10353/ this script demonstrates the usage of set(hash-able), a data structure that is much faster than list when you have two files, put the information from one file into a container, loop over the other file. https://divingintogeneticsandgenomics.com/archive/the-ngs-wikibook-a-dynamic-collaborative-online-training-effort-with-long-term/ Mon, 01 Jul 2013 18:22:00 -0700 https://divingintogeneticsandgenomics.com/archive/the-ngs-wikibook-a-dynamic-collaborative-online-training-effort-with-long-term/ The NGS wiki book http://en.wikibooks.org/wiki/Next_Generation_Sequencing_(NGS) http://intl-bib.oxfordjournals.org/content/early/2013/06/21/bib.bbt045.full for more details, refer to the link above. Nine simple rules to begin with NGS analysis RULE 1: DO NOT FEAR THE COMMAND LINE RULE 2: KNOW THE CONVENTIONS RULE 3: READ INTRODUCTORY REVIEWS RULE 4: START WITH QUALITY CHECKING RULE 5: PLAN FOR MISTAKES AND DOCUMENT WORKFLOW RULE 6: ALWAYS GET INFORMED AND GET HELP IF STUCK RULE 7: USE AN EFFICIENT INTEGRATIVE APPROACH RULE 8: AVOID REINVENTING THE WHEEL ( I mentioned this several times in my blog :) ) RULE 9: EDUCATION IS IMPORTANT https://divingintogeneticsandgenomics.com/archive/vcf-file-annotation-and-manipulation-tools/ Sat, 29 Jun 2013 05:14:00 -0700 https://divingintogeneticsandgenomics.com/archive/vcf-file-annotation-and-manipulation-tools/ VCF (variant calling format) file, as specified in 1000 Genome http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 is a common file to handle the variant information from the NGS sequencing data. many tools have been developed to manipulate, extract information with this format. some popular ones are: tabix http://samtools.sourceforge.net/tabix.shtml for fast indexing and extracting certain regions of the whole genome. VCFtools http://vcftools.sourceforge.net/ for many more manipulations including annotating, merging, concatenating, comparing etc. bedtools http://code.google.com/p/bedtools/ for genomic region overlapping calculation. https://divingintogeneticsandgenomics.com/archive/count-how-many-mapped-reads-in-a-bam-file/ Tue, 18 Jun 2013 19:09:00 -0700 https://divingintogeneticsandgenomics.com/archive/count-how-many-mapped-reads-in-a-bam-file/ I wanted to count how many mapped reads are in a ChIP-seq bam file downloaded from UCSC ECODE project. http://genome.ucsc.edu/ENCODE/downloads.html A quick google search: http://left.subtree.org/ and http://seqanswers.com/forums/showthread.php?t=891 The output from short read aligners like Bowtie and BWA is commonly stored in SAM/BAM format. When presented with one of these files a common first task is to calculate the total number of alignments (reads) captured in the file. In this post I show some examples for finding the total number of reads using samtools and directly from Java code. https://divingintogeneticsandgenomics.com/archive/fasta-q-file-manipulation-tool-readfq/ Tue, 11 Jun 2013 08:24:00 -0700 https://divingintogeneticsandgenomics.com/archive/fasta-q-file-manipulation-tool-readfq/ it is developed by Heng Li in princeton. https://github.com/lh3/readfq it is super fast and I will have a try. Readfq is a collection of routines for parsing the FASTA/FASTQ format. It seamlessly parses both FASTA and multi-line FASTQ with a simple interface. Readfq is first implemented in a single C header file and then ported to Lua, Perl and Python as a single function less than 50 lines. For users of scripting languages, I encourage to copy-and-paste the function instead of using readfq as a library. https://divingintogeneticsandgenomics.com/archive/remap-the-eland-sorted-txt-file-from-hg18-to-hg19/ Fri, 07 Jun 2013 13:51:00 -0700 https://divingintogeneticsandgenomics.com/archive/remap-the-eland-sorted-txt-file-from-hg18-to-hg19/ Sometime ago, I wrote a blog about how to convert Eland sorted.txt file to bam file convert Eland sorted.txt to bam file the original Eland file was mapped to hg18, but I want to remap it to hg19. essentially, I need a fastq file to map it by bowtie. A quick google search: http://seqanswers.com/forums/showthread.php?t=7061 so, the bamtools https://github.com/pezmaster31/bamtools (yes, github again...that's why I want to learn git a little bit, see my last post My first try on Git) https://divingintogeneticsandgenomics.com/archive/count-chip-seq-rna-seq-counts-at-features/ Sat, 01 Jun 2013 17:16:00 -0700 https://divingintogeneticsandgenomics.com/archive/count-chip-seq-rna-seq-counts-at-features/ several tools are: subread http://subread.sourceforge.net/ http://bioinf.wehi.edu.au/featureCounts/ Bedtools genomecov http://bedtools.readthedocs.org/en/latest/content/tools/genomecov.html HTseq: http://www-huber.embl.de/users/anders/HTSeq/doc/count.html Homer: http://biowhat.ucsd.edu/homer/ngs/quantification.html https://divingintogeneticsandgenomics.com/archive/a-very-good-introduction-of-ngs-analysis-in-the-homer-page/ Sat, 01 Jun 2013 16:48:00 -0700 https://divingintogeneticsandgenomics.com/archive/a-very-good-introduction-of-ngs-analysis-in-the-homer-page/ See the link here: http://biowhat.ucsd.edu/homer/basicTutorial/index.html Setting up your computing environment Next-generation sequencing analysis is a compute intensive process. Your average laptop is probably not up to the challenge. Most software is geared toward UNIX style operating systems, with large servers in mind. Typically, analysis algorithms will be distributed by researchers in one of three ways: Stand alone program to run in a UNIX environment (most common) Webserver where you upload data to be analyzed (i. https://divingintogeneticsandgenomics.com/archive/framework-for-evaluating-variant-detection-methods-comparison-of-aligners-and/ Mon, 13 May 2013 20:11:00 -0700 https://divingintogeneticsandgenomics.com/archive/framework-for-evaluating-variant-detection-methods-comparison-of-aligners-and/ http://bcbio.wordpress.com/ Blue Collar Bioinformatics Framework for evaluating variant detection methods: comparison of aligners and callers with 5 comments Variant detection and grading overview Developing pipelines for detecting variants from high throughput sequencing data is challenging due to rapidly changing algorithms and relatively low concordance between methods. This post will discuss automated methods providing evaluation of variant calls, enabling detailed diagnosis of discordant differences between multiple calling approaches. This allows us to: https://divingintogeneticsandgenomics.com/archive/0-based-and-1-based-coordinates/ Wed, 08 May 2013 19:14:00 -0700 https://divingintogeneticsandgenomics.com/archive/0-based-and-1-based-coordinates/ see the blog here: http://allaboutbioinfo.blogspot.com/search?updated-max=2011-08-18T16:56:00-07:00&max-results=7 ### There are two different coordinate base systems in different files: 0-base and 1-base. Different files use different base systems, and sometimes it causes confusions (especially when one tries to calculate the length of the region). Here I show the differences in two systems, and summarize several file formats that use both sytems.Bases in sequence positions 0-base system: the first base is 0. You represent a region as [a, b). https://divingintogeneticsandgenomics.com/archive/batch-converting-coordinates-to-sequences/ Tue, 30 Apr 2013 19:20:00 -0700 https://divingintogeneticsandgenomics.com/archive/batch-converting-coordinates-to-sequences/ If you have a file which contains the coordinates of the chromosomes ( let's say you have a bed https://genome.ucsc.edu/FAQ/FAQformat.html#format1 file from a ChIP-seq peak call) , how can you get the sequences of corresponding coordinates? ( you may need the sequences for motif analysis by MEME http://meme.nbcr.net/meme/intro.html )  a simple way is using the table browser in UCSC genome browser  Extracting sequence in batch from an assembly 
 Question: https://divingintogeneticsandgenomics.com/archive/cool-use-of-unix-paste-with-ngs-sequences-fastq-to-fasta/ Thu, 04 Apr 2013 19:29:00 -0700 https://divingintogeneticsandgenomics.com/archive/cool-use-of-unix-paste-with-ngs-sequences-fastq-to-fasta/ http://thegenomefactory.blogspot.com.au/search?updated-max=2012-09-08T12:11:00%2B10:00&max-results=7 Cool use of Unix paste with NGS sequences While browsing SeqAnswers.com today I came across a post where Uwe Appelt provided a couple of lines of Unix shell wizadry to solve some problem. What attracted my attention was the following: paste - - - - < in.fq | filter | tr "\t" "\n" > out.fq Now, I've done a reasonable amount of shell one-liners in my life, but I'd never seen this before. https://divingintogeneticsandgenomics.com/archive/trimming-adaptors-from-short-read-sequences/ Wed, 03 Apr 2013 09:49:00 -0700 https://divingintogeneticsandgenomics.com/archive/trimming-adaptors-from-short-read-sequences/ http://bcbio.wordpress.com/2009/08/09/trimming-adaptors-from-short-read-sequences/ Blue Collar Bioinformatics Trimming adaptors from short read sequences with 11 comments One common post processing step for next generation, or short read, sequencing is to remove adaptor sequences. Sample preparation for sequencing often involves selection of a particular fraction of interest like expressed genes, microRNAs, or copy number variant regions. Selecting, amplifying, and sequencing these regions can result in read sequences containing a common tag that needs to be identified and removed before continuing with other analysis steps like alignment to the genome.