Error in hybrid assembly occurred
Hi, When I run the assembly step, the following error in hybrid assembly occurred. How can I fix this? Thank you.
$impy assembly -m ${MG1} -m ${MG2} -m ${MG_single} -t ${MT1} -t ${MT2} -t ${MT_single} -o ${Dir_out}
.
.
.
(omitted)
.
.
.
rule megahit_assembly_from_unmapped:
input: Assembly/mt.r1.unmapped.fq, Assembly/mt.r2.unmapped.fq, Assembly/mt.se.unmapped.fq
output: Assembly/mt.megahit_unmapped.2/final.contigs.fa, Assembly/mt.megahit_unmapped.2.fa
wildcards: loop=2, type=mt
[x] Performing mt assembly step '2' using MEGAHIT
MEGAHIT v1.0.6
--- [Tue Apr 19 16:27:43 2022] Start assembly. Number of CPU threads 4 ---
--- [Tue Apr 19 16:27:43 2022] Available memory: 33675894784, used: 24000000000
--- [Tue Apr 19 16:27:43 2022] k list: 25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97,99 ---
--- [Tue Apr 19 16:27:43 2022] Converting reads to binaries ---
[read_lib_functions-inl.h : 209] Lib 0 (Assembly/mt.r1.unmapped.fq,Assembly/mt.r2.unmapped.fq): pe, 159143522 reads, 100 max length
[read_lib_functions-inl.h : 209] Lib 1 (Assembly/mt.se.unmapped.fq): se, 0 reads, 0 max length
[utils.h : 126] Real: 747.1210 user: 148.4221 sys: 71.0971 maxrss: 155120
--- [Tue Apr 19 16:40:10 2022] Extracting solid (k+1)-mers for k = 25 ---
--- [Tue Apr 19 16:54:03 2022] Building graph for k = 25 ---
--- [Tue Apr 19 16:54:08 2022] Assembling contigs from SdBG for k = 25 ---
--- [Tue Apr 19 16:54:22 2022] Local assembling k = 25 ---
--- [Tue Apr 19 16:56:46 2022] Extracting iterative edges from k = 25 to 29 ---
--- [Tue Apr 19 17:09:31 2022] Building graph for k = 29 ---
--- [Tue Apr 19 17:09:34 2022] Assembling contigs from SdBG for k = 29 ---
--- [Tue Apr 19 17:09:48 2022] Local assembling k = 29 ---
--- [Tue Apr 19 17:12:15 2022] Extracting iterative edges from k = 29 to 33 ---
--- [Tue Apr 19 17:25:23 2022] Building graph for k = 33 ---
--- [Tue Apr 19 17:25:27 2022] Assembling contigs from SdBG for k = 33 ---
--- [Tue Apr 19 17:25:43 2022] Local assembling k = 33 ---
--- [Tue Apr 19 17:28:13 2022] Extracting iterative edges from k = 33 to 37 ---
--- [Tue Apr 19 17:41:24 2022] Building graph for k = 37 ---
--- [Tue Apr 19 17:41:27 2022] Assembling contigs from SdBG for k = 37 ---
--- [Tue Apr 19 17:41:45 2022] Local assembling k = 37 ---
--- [Tue Apr 19 17:43:45 2022] Extracting iterative edges from k = 37 to 41 ---
--- [Tue Apr 19 17:55:27 2022] Building graph for k = 41 ---
--- [Tue Apr 19 17:55:31 2022] Assembling contigs from SdBG for k = 41 ---
--- [Tue Apr 19 17:55:48 2022] Local assembling k = 41 ---
--- [Tue Apr 19 17:57:48 2022] Extracting iterative edges from k = 41 to 45 ---
--- [Tue Apr 19 18:08:51 2022] Building graph for k = 45 ---
--- [Tue Apr 19 18:08:55 2022] Assembling contigs from SdBG for k = 45 ---
--- [Tue Apr 19 18:09:12 2022] Local assembling k = 45 ---
--- [Tue Apr 19 18:11:13 2022] Extracting iterative edges from k = 45 to 49 ---
--- [Tue Apr 19 18:21:23 2022] Building graph for k = 49 ---
--- [Tue Apr 19 18:21:27 2022] Assembling contigs from SdBG for k = 49 ---
--- [Tue Apr 19 18:21:44 2022] Local assembling k = 49 ---
--- [Tue Apr 19 18:23:45 2022] Extracting iterative edges from k = 49 to 53 ---
--- [Tue Apr 19 18:34:01 2022] Building graph for k = 53 ---
--- [Tue Apr 19 18:34:05 2022] Assembling contigs from SdBG for k = 53 ---
--- [Tue Apr 19 18:34:20 2022] Local assembling k = 53 ---
--- [Tue Apr 19 18:36:22 2022] Extracting iterative edges from k = 53 to 57 ---
--- [Tue Apr 19 18:45:32 2022] Building graph for k = 57 ---
--- [Tue Apr 19 18:45:35 2022] Assembling contigs from SdBG for k = 57 ---
--- [Tue Apr 19 18:45:48 2022] Local assembling k = 57 ---
--- [Tue Apr 19 18:47:52 2022] Extracting iterative edges from k = 57 to 61 ---
--- [Tue Apr 19 18:55:52 2022] Building graph for k = 61 ---
--- [Tue Apr 19 18:55:55 2022] Assembling contigs from SdBG for k = 61 ---
--- [Tue Apr 19 18:56:06 2022] Local assembling k = 61 ---
--- [Tue Apr 19 18:58:09 2022] Extracting iterative edges from k = 61 to 65 ---
--- [Tue Apr 19 19:05:34 2022] Building graph for k = 65 ---
--- [Tue Apr 19 19:05:36 2022] Assembling contigs from SdBG for k = 65 ---
--- [Tue Apr 19 19:05:45 2022] Local assembling k = 65 ---
--- [Tue Apr 19 19:07:48 2022] Extracting iterative edges from k = 65 to 69 ---
--- [Tue Apr 19 19:15:06 2022] Building graph for k = 69 ---
--- [Tue Apr 19 19:15:08 2022] Assembling contigs from SdBG for k = 69 ---
--- [Tue Apr 19 19:15:17 2022] Local assembling k = 69 ---
--- [Tue Apr 19 19:17:18 2022] Extracting iterative edges from k = 69 to 73 ---
--- [Tue Apr 19 19:23:38 2022] Building graph for k = 73 ---
--- [Tue Apr 19 19:23:40 2022] Assembling contigs from SdBG for k = 73 ---
--- [Tue Apr 19 19:23:46 2022] Local assembling k = 73 ---
--- [Tue Apr 19 19:26:08 2022] Extracting iterative edges from k = 73 to 77 ---
--- [Tue Apr 19 19:31:26 2022] Building graph for k = 77 ---
--- [Tue Apr 19 19:31:28 2022] Assembling contigs from SdBG for k = 77 ---
--- [Tue Apr 19 19:31:32 2022] Local assembling k = 77 ---
--- [Tue Apr 19 19:33:29 2022] Extracting iterative edges from k = 77 to 81 ---
--- [Tue Apr 19 19:38:04 2022] Building graph for k = 81 ---
--- [Tue Apr 19 19:38:06 2022] Assembling contigs from SdBG for k = 81 ---
--- [Tue Apr 19 19:38:09 2022] Local assembling k = 81 ---
--- [Tue Apr 19 19:40:01 2022] Extracting iterative edges from k = 81 to 85 ---
--- [Tue Apr 19 19:43:38 2022] Building graph for k = 85 ---
--- [Tue Apr 19 19:43:40 2022] Assembling contigs from SdBG for k = 85 ---
--- [Tue Apr 19 19:43:42 2022] Local assembling k = 85 ---
--- [Tue Apr 19 19:45:34 2022] Extracting iterative edges from k = 85 to 89 ---
--- [Tue Apr 19 19:48:30 2022] Building graph for k = 89 ---
--- [Tue Apr 19 19:48:31 2022] Assembling contigs from SdBG for k = 89 ---
--- [Tue Apr 19 19:48:33 2022] Local assembling k = 89 ---
--- [Tue Apr 19 19:50:22 2022] Extracting iterative edges from k = 89 to 93 ---
--- [Tue Apr 19 19:52:38 2022] Building graph for k = 93 ---
--- [Tue Apr 19 19:52:39 2022] Assembling contigs from SdBG for k = 93 ---
--- [Tue Apr 19 19:52:40 2022] Local assembling k = 93 ---
--- [Tue Apr 19 19:54:39 2022] Extracting iterative edges from k = 93 to 97 ---
--- [Tue Apr 19 19:56:32 2022] Building graph for k = 97 ---
--- [Tue Apr 19 19:56:33 2022] Assembling contigs from SdBG for k = 97 ---
--- [Tue Apr 19 19:56:34 2022] Local assembling k = 97 ---
--- [Tue Apr 19 19:58:26 2022] Extracting iterative edges from k = 97 to 99 ---
--- [Tue Apr 19 20:00:00 2022] Building graph for k = 99 ---
--- [Tue Apr 19 20:00:00 2022] Assembling contigs from SdBG for k = 99 ---
--- [Tue Apr 19 20:00:01 2022] Merging to output final contigs ---
--- [STAT] 66 contigs, total 24000 bp, min 203 bp, max 599 bp, avg 364 bp, N50 358 bp
--- [Tue Apr 19 20:00:02 2022] ALL DONE. Time elapsed: 12739.113557 seconds ---
5 of 15 steps (33%) done
rule idba_hybrid_assembly_1:
input: Preprocessing/mg.r1.preprocessed.fq, Preprocessing/mg.r2.preprocessed.fq, Preprocessing/mg.se.preprocessed.fq, Preprocessing/mt.r1.preprocessed.fq, Preprocessing/mt.r2.preprocessed.fq, Preprocessing/mt.se.preprocessed.fq, Assembly/mt.megahit_preprocessed.1/final.contigs.fa, Assembly/mt.megahit_unmapped.2/final.contigs.fa
output: Assembly/mgmt.idba_hybrid.1.fa
[x] Performing first hyrbid assembly step using IDBA
[x] Interleave MG and MT fastq files
[x] Join MG and MT interleaved fasta files
[x] Concatenate MT contigs, MT and MG single end files
number of threads 4
bash: line 14: 12672 Killed idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
Error in job idba_hybrid_assembly_1 while creating output file Assembly/mgmt.idba_hybrid.1.fa.
RuleException:
CalledProcessError in line 16 of /home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules:
Command '
echo "[x] Performing first hyrbid assembly step using IDBA"
echo "[x] Interleave MG and MT fastq files"
TMPD=$(mktemp -d -t --tmpdir=/home/imp/output/tmp "XXXXXX")
fq2fa --merge Preprocessing/mg.r1.preprocessed.fq Preprocessing/mg.r2.preprocessed.fq $TMPD/merged_MG.fa
fq2fa --merge Preprocessing/mt.r1.preprocessed.fq Preprocessing/mt.r2.preprocessed.fq $TMPD/merged_MT.fa
echo "[x] Join MG and MT interleaved fasta files"
cat $TMPD/merged_MG.fa $TMPD/merged_MT.fa > $TMPD/merged.fa
echo "[x] Concatenate MT contigs, MT and MG single end files"
cat <(cat Assembly/mt.megahit_preprocessed.1/final.contigs.fa Assembly/mt.megahit_unmapped.2/final.contigs.fa | awk '/^>/{print ">contig_MT_" ++i; next}{print}') <(cat Preprocessing/mg.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') <(cat Preprocessing/mt.se.preprocessed.fq | sed -n '1~4s/^@/>/p;2~4p') > $TMPD/MT_contigs-MG_MT.SE.fa
idba_ud -r $TMPD/merged.fa -l $TMPD/MT_contigs-MG_MT.SE.fa -o $TMPD --mink 25 --maxk 99 --step 4 --num_threads 4 --similar 0.98 --pre_correction
mv $TMPD/contig.fa Assembly/mgmt.idba_hybrid.1.fa
rm -rf $TMPD
' returned non-zero exit status 137
File "/home/imp/code/rules/Assembly/hybrid/idba.hybrid.rules", line 16, in __rule_idba_hybrid_assembly_1
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
Will exit after finishing currently running jobs.
/home/imp/data/1018_metagenome_1.fastq => Preprocessing/mg.r1.fq
/home/imp/data/1018_metagenome_2.fastq => Preprocessing/mg.r2.fq
/home/imp/data/1018_metagenome_single.fastq => Preprocessing/mg.se.fq
/home/imp/data/1018_transcriptome_1.fastq => Preprocessing/mt.r1.fq
/home/imp/data/1018_transcriptome_2.fastq => Preprocessing/mt.r2.fq
/home/imp/data/1018_transcripptome_single.fastq => Preprocessing/mt.se.fq
symlink mg.r1.fq => Preprocessing/mg.r1.preprocessed.fq
symlink mg.r2.fq => Preprocessing/mg.r2.preprocessed.fq
symlink mg.se.fq => Preprocessing/mg.se.preprocessed.fq
symlink mt.r1.fq => Preprocessing/mt.r1.preprocessed.fq
symlink mt.r2.fq => Preprocessing/mt.r2.preprocessed.fq
symlink mt.se.fq => Preprocessing/mt.se.preprocessed.fq
Exiting because a job execution failed. Look above for error message