do not put all csv string in memory

[LuyiTian]

currently, in the sc_demultiplex step, the program will keep all the reads in memory and then open CSV and write to them one by one (because we cant open too many files at the same time). it will become a problem when we have a very large dataset.

to solve this. we need to write to CSV file during we process the bam file. it can be that every 100000 reads we trigger a write_to_csv thing and clear the cache.

[cpattaroni]

I currently have a massive dataset (6 billion reads to start with). Memory was an issue but I managed to get it running with 900Gb RAM... It completed the task, got to the next line but the output was completely empty. Any idea why?

[cpattaroni]

I confirm there is an issue for big files (starting with 6 billions reads). Even with enough RAM, it is not writing .csv file properly.

[LuyiTian]

@cpattaroni what protocol did you use? if it is a combination of multiple run you could split the data and run each one separately.

[cpattaroni]

@LuyiTian It's QIAseq UPX 3' Transcriptome, not very common. It worked fine a year ago with a small run and previous versions of R and scPipe.

I've tried splitting it into 4, still not writing it. I've stopped using it for this reason, trying with umi-tools at the moment.