The first full quarterly HCP Data release is now Cloud accessible for participants who have registered and agreed to the Open Access Data Use Terms.

HBM Hackathon participants will have the opportunity for high speed access to the first full quarterly HCP Data release via Amazon Web Services. All data available from the HCP Connectome-in-a-Box can now be downloaded directly from S3, Amazon’s Simple Storage Service.

This post is organized into three sections:

1. Description of the HCP Q1 Release
2. Getting Authorized to Access the HCP Data
3. Accessing HCP Data on AWS

Description of the HCP Q1 Release

The Q1 data release consists of multimodal MRI data collected from 68 healthy young adults who were scanned in the fall of 2012. These include all 12 subjects from our Initial Data Release. All 3T MRI scan data is included: Structural, Functional (resting state and task) and Diffusion. Behavioral data collected on all subjects is also included, with the exception of sensitive restricted-access data. Please see the Full Release Documentation for further details.

Getting Authorized to Access the HCP Data

There are a few steps you will need to take to get authorization to access the HCP data hosted on AWS.

You need to:

1. Complete the HCP Registration and agree to the HCP Open Access Data Use Terms
2. Complete the HBM Hackathon Registration
3. Create an Amazon Web Services Account
4. Email hbm.hackathon@gmail.com with your HCP and AWS email address(es)

A foreword from David van Essen:

For distribution to all investigators interested in using HCP Connectome-in-a-Box data.

IMPORTANT NOTICE to investigators wanting to use HCP datasets available on Connectome-in-a-Box hard drives.

HCP’s Connectome-in-a-Box provides imaging data from the Open Access dataset. Before using any of these data for research, you and all other investigators using the data are required to register and agree to the Open Access Data Use Terms. This includes agreeing to comply with institutional rules and regulations. This may mean that you need your research to be approved or declared exempt by a committee that oversees research on human subjects (e.g., your IRB or Ethics Committee). The released HCP data are not considered de-identified, insofar as certain combinations of HCP Restricted Data (available through a separate process) might allow identification of individuals. Different committees operate under different national, state and local laws and may interpret regulations differently, so it is important to ask about this. If needed and upon request, the HCP will provide a certificate stating that you have accepted the HCP Open Access Data Use Terms.

Sincerely,
David C. Van Essen (PI), for the WU-Minn HCP Consortium
May 1, 2013

Accessing HCP Data on AWS

Amazon Web Services is hosting the HCP Q1 Data Release as part of its Public Data Sets on AWS program, which will enable HBM Hackathon participants to get rapid access to the HCP data. Since the data is hosted on S3 in an uncompressed format, participants can download data in parallel using tools like s3cmd-modification.

You can use tools like s3cmd to list the contents of a directory on S3, get individual files or sync full directories. You will also be able to download directly to your personal/work computer or to a machine on the EC2 Cloud like the NITRC Computational Environment.

Getting Credentials

Ready to start downloading? Make sure you’ve completed the HCP registration and created an account with AWS.

Next, send an email to hbm.hackathon@gmail.com with your HCP and AWS email address(es).

We will use your HCP and AWS email address(es) to:

1. verify that you have agreed to the Open Access Data Use Agreement
2. grant you access to the HCP data on AWS.

Once verified, you will receive an email confirming that you have access to the HCP data on AWS.

Configuring your system

To access the data you’ll want to install s3cmd-modification, which will enable you to explore the HCP Data Directory on Amazon and will allow rapid parallel downloading (s3cmd-modification install instructions).

After you install s3cmd, you need to configure it with your AWS public and secret keys, located in AWS Security Credentials.

:~ s3cmd --configureEnter new values or accept defaults in brackets with Enter.Refer to user manual for detailed description of all options.Access key and Secret key are your identifiers for Amazon S3Access Key []: <your-access-key>Secret Key []: <your-secret-key>... 

Take s3cmd out for a test drive…

List subject directories:

:~ s3cmd ls s3://hcp.aws.amazon.com/q1/DIR   s3://hcp.aws.amazon.com/q1/100307/DIR   s3://hcp.aws.amazon.com/q1/103515/DIR   s3://hcp.aws.amazon.com/q1/111312/...DIR   s3://hcp.aws.amazon.com/q1/937160/2013-04-17 06:57         0   s3://hcp.aws.amazon.com/q1/

Get a directory in parallel:

:~ s3cmd --parallel --workers=16 get --recursive s3://hcp.aws.amazon.com/q1/100307/T1w T1wFile s3://hcp.aws.amazon.com/q1/100307/T1w/Bi... started [2 of 52]File s3://hcp.aws.amazon.com/q1/100307/T1w/Na... started [3 of 52]...File s3://hcp.aws.amazon.com/q1/100307/T1w/T2... saved as 'T1w/T1w/T2w_acpc_dc_restore.nii.gz' (67855816 bytes in 143.3 seconds, 462.31 kB/s)File s3://hcp.aws.amazon.com/q1/100307/T1w/Bi... saved as 'T1w/T1w/BiasField_acpc_dc.nii.gz' (65076318 bytes in 180.3 seconds, 352.50 kB/s)

If everything checks out, you are ready to get hacking!