Hi Collin,

Regarding (1) yes, the number of jobs is actually manually set by users when they run some process using parallelization options in CONN, and all jobs are always fixed to use a single core. In particular when running some process/analysis a users choose N, the total number of jobs to submit, and CONN will divide the total number of subjects M in a project across those jobs so that each job uses a single core to processes approximately M/N subjects (see https://web.conn-toolbox.org/resources/h... for details)

And regarding (2), it's hard to quantify precisely but I imagine one could expect to have from a few tens to a few hundreds files per subject being created in a typical project, depending on the type of data and analysis being run. 

Hope this helps

Alfonso

Originally posted by Collin Wilson:

Hello,

I am an HPC analyst helping a user optimize their HPC workflow with CONN. There are two main issues:

1. Their parallel configuration seems to be lauching a single core job for each subject; this is not ideal as it puts stress on the schedule and they end up waiting for jobs that take <10m to complete. Can CONN jobs be configured to say launch X jobs each with Y cores processing Z subjects?
   
2. The output produces hundreds of thousands of files - can these results files be consolidated somehow? Users have a file number quota due to filesystem limitations.
   

Many thanks for your assistance!

Collin