Hello,

I am an HPC analyst helping a user optimize their HPC workflow with CONN. There are two main issues:

1. Their parallel configuration seems to be lauching a single core job for each subject; this is not ideal as it puts stress on the schedule and they end up waiting for jobs that take <10m to complete. Can CONN jobs be configured to say launch X jobs each with Y cores processing Z subjects?
   
2. The output produces hundreds of thousands of files - can these results files be consolidated somehow? Users have a file number quota due to filesystem limitations.
   

Many thanks for your assistance!

Collin

I should also mention we're using a slurm scheduler.

Hi Collin,

Regarding (1) yes, the number of jobs is actually manually set by users when they run some process using parallelization options in CONN, and all jobs are always fixed to use a single core. In particular when running some process/analysis a users choose N, the total number of jobs to submit, and CONN will divide the total number of subjects M in a project across those jobs so that each job uses a single core to processes approximately M/N subjects (see https://web.conn-toolbox.org/resources/h... for details)

And regarding (2), it's hard to quantify precisely but I imagine one could expect to have from a few tens to a few hundreds files per subject being created in a typical project, depending on the type of data and analysis being run. 

Hope this helps

Alfonso

Originally posted by Collin Wilson:

Hello,

I am an HPC analyst helping a user optimize their HPC workflow with CONN. There are two main issues:

1. Their parallel configuration seems to be lauching a single core job for each subject; this is not ideal as it puts stress on the schedule and they end up waiting for jobs that take <10m to complete. Can CONN jobs be configured to say launch X jobs each with Y cores processing Z subjects?
   
2. The output produces hundreds of thousands of files - can these results files be consolidated somehow? Users have a file number quota due to filesystem limitations.
   

Many thanks for your assistance!

Collin

Thank you, this is helpful.

Follow up: Does the number of files created depend in any way on the number of parallel processes?

Hi Collin,

The number of files created in any given process/anaysis will be exacty the same independent on the number of jobs in charge of running that process/analysis (well, with the exception of course of a few log- and control- files which are created separately by each individual job and are stored in the {connproject}.qlog directory)

Best

Alfonso 

Originally posted by Collin Wilson:

Thank you, this is helpful.

Follow up: Does the number of files created depend in any way on the number of parallel processes?