https://divingintogeneticsandgenomics.com/tags/big-data/ Recent content in Big data on Chatomics Hugo -- gohugo.io en-us © 2018 Ming 'Tommy' Tang Sun, 02 Oct 2016 19:39:00 -0700 https://divingintogeneticsandgenomics.com/archive/cutting-out-500-columns-from-a-26g-file-using-command-line/ Sun, 02 Oct 2016 19:39:00 -0700 https://divingintogeneticsandgenomics.com/archive/cutting-out-500-columns-from-a-26g-file-using-command-line/ I have a 26 G tsv file with several thousand columns. I want to extract 500 columns from it based on the column names in another file. How should I do it? Reading into R may take forever, although one may recommend using data.table to fread in the data to save some time. However, R is notorious for having to read in the data into memory. 26G is very big and my desktop does not have that power. https://divingintogeneticsandgenomics.com/archive/materials-for-the-1st-summer-institute-in-statistics-for-big-data/ Tue, 14 Jul 2015 21:56:00 -0700 https://divingintogeneticsandgenomics.com/archive/materials-for-the-1st-summer-institute-in-statistics-for-big-data/ I am here in the University of Washington for the 1st Summer Institute in Statistics for Big Data. This 2 weeks' summer institute covers from getting big data, visualizing big biomedical data, machine learning to reproducible research. Check the website here: http://www.biostat.washington.edu/suminst/sisbid/modules All the studying materials are on github: https://github.com/SISBID Free ebook for statistical learning: An introduction to statistical learning