Preprocessing: more efficient (de)compression
- preparing input data: pipeline can fail because of memory issues when having large gzipped files
- compressing intermediate FASTQ files
Use pigz
and pbzip2
instead of gzip
and bzip2
, and python
's gzip
package