Fail to open existing file

[elcortegano]

Hi, manta is returning an error that cannot open a file (that exists), but that I think might relate to the number of samples used, or probably something else. When I run manta with the desired number of BAMs (between 14 and 30), it will break. A cut from the error log is shown below.

[TaskRunner:generateCandidateSV_0000] Task initiated on local node
[TaskRunner:generateCandidateSV_0000] [WARNING] Task: 'generateCandidateSV_0000' failed but qualifies for retry. Total task failures (including this one): 1. Task command: '/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam'
[TaskRunner:generateCandidateSV_0000] Retrying task: 'generateCandidateSV_0000'. Total prior task failures: 1
[TaskRunner:generateCandidateSV_0000] Task initiated on local node
[TaskRunner:generateCandidateSV_0000] [WARNING] Task: 'generateCandidateSV_0000' failed but qualifies for retry. Total task failures (including this one): 2. Task command: '/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam'
[TaskRunner:generateCandidateSV_0000] Retrying task: 'generateCandidateSV_0000'. Total prior task failures: 2
[TaskRunner:generateCandidateSV_0000] Task initiated on local node
[TaskManager] [ERROR] Failed to complete command task: 'generateCandidateSV_0000' launched from master workflow, error code: 1, command: '/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam'
[TaskManager] [ERROR] [generateCandidateSV_0000] Error Message:
[TaskManager] [ERROR] [generateCandidateSV_0000] Last 6 stderr lines from task (of 6 total lines):
[TaskManager] [ERROR] [generateCandidateSV_0000] [E::hts_open_format] Failed to open file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam
[TaskManager] [ERROR] [generateCandidateSV_0000] FATAL_ERROR: Failed to open SAM/BAM/CRAM file for reading: '/localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam'
[TaskManager] [ERROR] [generateCandidateSV_0000] cmdline:	/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam
[TaskManager] [ERROR] [generateCandidateSV_0000] version:	1.6.0
[TaskManager] [ERROR] [generateCandidateSV_0000] buildTime:	2019-06-28T22:06:27.673004Z
[TaskManager] [ERROR] [generateCandidateSV_0000] compiler:	g++-6.3.1
[TaskManager] [ERROR] Shutting down task submission. Waiting for remaining tasks to complete.
[WorkflowRunner] [ERROR] Workflow terminated due to the following task errors:
[WorkflowRunner] [ERROR] Failed to complete command task: 'generateCandidateSV_0000' launched from master workflow, error code: 1, command: '/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam'
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] Error Message:
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] Last 6 stderr lines from task (of 6 total lines):
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] [E::hts_open_format] Failed to open file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] FATAL_ERROR: Failed to open SAM/BAM/CRAM file for reading: '/localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam'
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] cmdline:	/localdisk/home/elopezc/manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 72 --align-stats /home/elopezc/manta/Cincerta/workspace/alignmentStats.xml --graph-file /home/elopezc/manta/Cincerta/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /localdisk/data/chlamydomonas/c_incerta_MA/reference_genome/Chlamydomonas_incerta.V3.fa --candidate-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /home/elopezc/manta/Cincerta/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /home/elopezc/manta/Cincerta/workspace/chromDepth.txt --edge-runtime-log /home/elopezc/manta/Cincerta/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.xml --edge-stats-report /home/elopezc/manta/Cincerta/results/stats/svCandidateGenerationStats.tsv --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L01.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L02.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L03.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L04.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L05.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L06.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L07.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L08.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L09.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L10.bam --align-file /localdisk/data/chlamydomonas/c_incerta_MA/MA_lines_alignment/markdup_C-incerta-L11.bam
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] version:	1.6.0
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] buildTime:2019-06-28T22:06:27.673004Z
[WorkflowRunner] [ERROR] [generateCandidateSV_0000] compiler:	g++-6.3.1

It can be seen that this is indicating a FATAL ERROR opening one of the BAM files. However, this BAM does not give any problem with other software, or even with Manta when this BAM is used as single file for input.

I'm running Manta 1.6.0 using the recommended binary file (manta-1.6.0.centos6_x86_64).

Thanks,
Eugenio

[zhk2017]

unlimit the number of open files, if centos system change this setting use ulimit -HSn 65535.

[elcortegano]

Apparently, using the default with all cores is what gave trouble. Including option -j 4 works fine.