https://divingintogeneticsandgenomics.com/tags/awk/ Recent content in Awk on Chatomics Hugo -- gohugo.io en-us © 2018 Ming 'Tommy' Tang Tue, 25 Aug 2015 13:36:00 -0700 https://divingintogeneticsandgenomics.com/archive/sort-with-header-maintained/ Tue, 25 Aug 2015 13:36:00 -0700 https://divingintogeneticsandgenomics.com/archive/sort-with-header-maintained/ We usually want to maintain header when we do something for the body of the text. You can find a body function here which can only accept streams and assumes only one line header. Use subshells and awk can do the same job and potentially more flexible. View this gist on GitHub The original credits go to Aaron Quinlan. see a gist below to sort vcf files in natural chromosome order :chr1 chr2 chr3. https://divingintogeneticsandgenomics.com/archive/using-awk-to-format-the-gene-expression-file-to-gct-format-for-gene-set/ Tue, 19 May 2015 13:42:00 -0700 https://divingintogeneticsandgenomics.com/archive/using-awk-to-format-the-gene-expression-file-to-gct-format-for-gene-set/ After one runs microarray or RNA-seq analysis, usually he would do a Gene Set Enrichment Analysis (GSEA) analysis. There are many tools to use. One of the most commonly used one is GSEA developed in Broad Institute. It requires four data files to be loaded: 1. Expression dataset in res, gct, pcl or txt format 2. Phenotype labels in cls format 3. Gene sets in gmx or gmt formt 4. Chip annotations https://divingintogeneticsandgenomics.com/archive/using-awk-to-join-two-files-based-on-several-columns/ Thu, 14 May 2015 14:35:00 -0700 https://divingintogeneticsandgenomics.com/archive/using-awk-to-join-two-files-based-on-several-columns/ I was reading a thread on stackoverflow and found that this post was very interesting. I will go through the problem and the awk solution. Again, Awk is awesome! Note: It looks a bit messy when I copied the ipython notebook directly into Blogger. Ideally, one can write the blog using ipython notebook directly, but it does not work well with Blogger. see here http://blog.fperez.org/2012/09/blogging-with-ipython-notebook.html I do not want to use other blogging platforms for now. https://divingintogeneticsandgenomics.com/archive/everyone-will-write-a-blog-post-on-dplyr/ Fri, 26 Sep 2014 12:28:00 -0700 https://divingintogeneticsandgenomics.com/archive/everyone-will-write-a-blog-post-on-dplyr/ dply is a nice R package to manipulate big data. There are several good tutorials I came across: http://www.dataschool.io/dplyr-tutorial-for-faster-data-manipulation-in-r/ http://stat545-ubc.github.io/block009_dplyr-intro.html http://stat545-ubc.github.io/bit001_dplyr-cheatsheet.html http://gettinggeneticsdone.blogspot.com/2014/08/do-your-data-janitor-work-like-boss.html I followed the example in the early edition of Bioinformatics Data Skills on the dplyr part and put a gist below. The example used dplyr to manipulate and summarize the human annotation gff file downloaded here http://useast.ensembl.org/info/data/ftp/index.html. I downloaded the ensemble 74 build which is different from the 75 build in the example of the book. https://divingintogeneticsandgenomics.com/archive/manipulate-gtf-file-with-awk/ Tue, 28 Jan 2014 08:57:00 -0800 https://divingintogeneticsandgenomics.com/archive/manipulate-gtf-file-with-awk/ AWK is awesome! or I should say linux is awesome! I had a post for linux one-liners here useful unix one liners for bioinformatics Today, I came across a blog series about GTF file manipulation using AWK. I feel it is cool, it teaches awk using a practical example. you will see the beauty of linux commands! here are the links: http://reasoniamhere.com/2013/09/16/awk-gtf-how-to-analyze-a-transcriptome-like-a-pro-part-1/ http://reasoniamhere.com/2013/09/17/awk-gtf-how-to-analyze-a-transcriptome-like-a-pro-part-2/ http://reasoniamhere.com/2013/09/18/awk-gtf-how-to-analyze-a-transcriptome-like-a-pro-part-3/ Enjoy! https://divingintogeneticsandgenomics.com/archive/useful-unix-one-liners-for-bioinformatics/ Thu, 24 Oct 2013 11:23:00 -0700 https://divingintogeneticsandgenomics.com/archive/useful-unix-one-liners-for-bioinformatics/ I have been using ubuntu 12.04LTS for one year, and I really love it and do not look back at windows at all. Currently, I have dual booting system for linux and windows, but now I barely fire the windows. The power of Linux comes from, at least in one aspect, its command line. As I have stated before, in the field of bioinformatics, we spend a lot of time in reformatting, cleaning, extracting the data. https://divingintogeneticsandgenomics.com/archive/group-the-same-id-to-the-same-line-by-python-use-awk-to-do-sql-like-groupby-and/ Fri, 18 Oct 2013 12:01:00 -0700 https://divingintogeneticsandgenomics.com/archive/group-the-same-id-to-the-same-line-by-python-use-awk-to-do-sql-like-groupby-and/ I was on Seqanswer forum and found someone asked a question: http://seqanswers.com/forums/showthread.php?p=119263&posted=1#post119263 "Hi guys, i hv got a GO file for my differentially expressed genes file, it goes like: FBgn00001 GO:0016301 [Name:****(annotation)] FBgn00002 GO:0016301 [Name:****(annotation)] FBgn00003 GO:0016301 [Name:****(annotation)] FBgn00004 GO:0003700 [Name:****(annotation)] FBgn00004 GO:0009651 [Name:****(annotation)] FBgn00004 GO:0006355 [Name:****(annotation)] FBgn00005 GO:0009556 [Name:****(annotation)] FBgn00005 GO:0005515 [Name:****(annotation)] FBgn00005 GO:0080019 [Name:****(annotation)] FBgn00005 GO:0016563 [Name:****(annotation)] FBgn00005 GO:0016627 [Name:****(annotation)] FBgn00006 GO:0003700 [Name:****(annotation)] FBgn00006 GO:0010018 [Name:****(annotation)] now i want to use WEGO ,so i need to convert it like: https://divingintogeneticsandgenomics.com/archive/awk-command-to-count-total-unique-and-the-most-abundant-read-in-a-fastq-file/ Thu, 16 May 2013 08:11:00 -0700 https://divingintogeneticsandgenomics.com/archive/awk-command-to-count-total-unique-and-the-most-abundant-read-in-a-fastq-file/ see post here: http://gettinggeneticsdone.blogspot.com/2012/04/awk-command-to-count-total-unique-and.html Awk Command to Count Total, Unique, and the Most Abundant Read in a FASTQ file I was reading through a paper on comparative ChIP-Seq when I found this awk gem that lets you get some very basic stats very quickly on next generation sequencing reads. To use, simply cat the fastq file (or gunzip -c) and pipe that to this awk command: cat myfile.fq | awk '((NR-2)%4==0){read=$1;total++;count[read]++}END{for(read in count){if(! https://divingintogeneticsandgenomics.com/archive/awk-for-simple-text-manipulation/ Mon, 22 Apr 2013 18:45:00 -0700 https://divingintogeneticsandgenomics.com/archive/awk-for-simple-text-manipulation/ let's say you have a bed file (tab delimited): tommy@tommy-ThinkPad-T420:~$ cat file.bed chr1 100 302 chr2 600 901 chr3 250 383 you want to calculate the average peak length of this file: tommy@tommy-ThinkPad-T420:~$ cat file.bed | awk '{print $3-$2}'| awk '{sum+=$0} END {print "Average= " sum/NR}' Average= 212 if you calculate the middle point of the peak, $2+ ($3- $2)/2 and you get a float, you want to round the column: