gatk MarkDuplicatesSpark 内存溢出

anneng

gatk MarkDuplicatesSpark -I ${res}/${sample_id}_aligned_reads.sam
-M ${res}/${sample_id}_dedup_metrics.txt -O ${res}/${sample_id}_sorted_dedup_reads.bam

https://gatk.broadinstitute.org/hc/en-us/community/posts/360058452072-MarkDuplicatesSpark-consumes-enormous-amount-of-RAM

gatk MarkDuplicatesSpark \
            -I input.bam \
            -O marked_duplicates.bam \
            -M marked_dup_metrics.txt \
            --conf 'spark.executor.instances=10' \
            --conf 'spark.executor.cores=10' \
            --conf 'spark.executor.memory=8G'

java.lang.OutOfMemoryError: GC overhead limit exceeded
21/08/23 13:13:08 ERROR SparkContext: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "stop-spark-context" java.lang.OutOfMemoryError: GC overhead limit exceeded
21/08/23 14:32:06 INFO SparkContext: SparkContext already stopped.
14:32:06.038 INFO MarkDuplicatesSpark - Shutting down engine
[August 23, 2021 2:32:06 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 497.40 minutes.
Runtime.totalMemory()=29172957184

A USER ERROR has occurred: Couldn't write file /data/result/X2_sorted_dedup_reads.bam because writing failed with exception java.lang.OutOfMemoryError: Java heap space

Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--java-options '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.
21/08/23 14:32:06 INFO ShutdownHookManager: Shutdown hook called
21/08/23 14:32:06 INFO ShutdownHookManager: Deleting directory /tmp/spark-ee1e0044-6489-40a8-82c1-8474c9706ef1/userFiles-ac9b0eff-c752-4128-984a-76a953546dc4
21/08/23 14:32:06 INFO ShutdownHookManager: Deleting directory /tmp/spark-ee1e0044-6489-40a8-82c1-8474c9706ef1