Dear Dr. Zalesky,

Thank you to you and your collegues for the wonderful tools and toolboxes you have given the neuroscientific research community!

I've been working with the NBS toolbox for most of the day, and while I have made great progress in understanding what I am doing, I'm still unsure about which design matrix to use (I've made quite a few).
The help section for statistical model was very useful, but after creating several different matrices and analyses, I'm still unsure which is correct.

Simply put, I have 48 matrices belonging to 12 participants (within subjects design). They were tested on 2 seperate days (Placebo vs Drug) and had two resting state scans on each day (No music vs Music). The order of the matrices (90x90x48) is as follows:


S01 Placebo No-music (0 0)
till
S12 Placebo No-music (0 0)

S01 Placebo Music  (0 1)
till
S12 Placebo Music (0 1)

S01 Drug No-music (1 0)
till
S12 Drug No-music (1 0)

S01 Drug Music (1 1)
till
S12 Drug Music (1 1)

I am interested in the main effects of drug and music, but more specifically in the interaction effect between drug and music (and subsequently visualizing this in BrainNet). Using FDR there are barely any suriving edges, and the NBS seems perfect for situations like these.

My main issue has been the design matrix. As pictured above, I initially had a simple 48x2 with 1s and 0s (and later 1 and -1), but was consistently getting 1000s of significant edges for certain thresholds (so clearly not looking at the specific interaction). Could you tell me the dimensions of the required/recommended design matrix? I've attempted a 48x3 for A, B and AxB, 48x5 for A1, A2, B1, B2, AB, but am confused about the requirement of a column for each participant?

If someone here could provide me with some tips/pointers about the required dimensions for the matrix, the interaction contrast, and how the first line in the matrxi would look, I'm sure I can figure out the rest through deductive reasoning. Any help would be greatly appericiated.

Sincerely,

David

ps. Would you recommend using raw R values or Fisher Z transformed R values?

Hi David,what you have is a 2 x 2 ANOVA with repeated measure on both factors. So you will need to use exchange blocks in your design to account for the repeated measures.

Let's assume you only have two subjects (you can easily generalize my below suggestions to 12 subjects). Let's use P and D to denote placebo and drug. Let's use M and N to denote music and no music. So "PN1" means the no music placebo condition of Subject 1. Your design matrix should look like this:

PN1 0 0 0 1 0
PN2 0 0 0 0 1
PM1 0 1 0 1 0
PM2 0 1 0 0 1
DN1 1 0 0 1 0
DN2 1 0 0 0 1
DM1 1 1 1 1 0
DM2 1 1 1 0 1

The 1st column is the main effect of drug. 2nd column is main effect of music. 3rd column is the interaction between music and drug. Column 4 and 5 is the mean of subject 1 and 2, respectively.

You should be able to extend this design matrix for 12 subjects. Note that if you have 12 subjects, you should have 15 columns in total and 12 rows. 1 column for each subject  + 2 main effects + 1 interaction = 15 columns.

To test for main effect of drug, contrast is: [1 0 0 0 0] or [-1 0 0 0 0]

Main effect of music: [0 1 0 0 0] or [0 -1 0 0 0]

Interaction: [0 0 1 0 0] or [0 0 -1 0 0]

Choose: "t-test"

The exchange block should be:
[1 2 1 2 1 2 1 2]

So I have given you all the information for the case of 2 subjects. Now you should be able to generalize this to the case of 12 subjects.

Let me now if you need help to generalize from 2 to 12 subjects.


Using r or r-to-z usually makes no difference. Reviewers might bring it up - so you might as well use r-to-z.

Andrew




Originally posted by David de Wide:

Hey Andrew,

Thank you for your quick and detailed reply.

Based on your suggestions I have contructed the 48x15 design matrix (included as attachement), contrasts and exchange blocks. As a sanity check, could you confirm that this is correct?

The contrasts for the interaction effect would be [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ] or [0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 ]. What is the exact difference between these two versions of the contrast?

The exhange blocks would be [1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12]

At a range of thresholds there appear to be no changes for the interaction effect with a lower p value than 0.2. 

I had an additional question regarding contrasts and network analysis, given your expertise. Is there a way to look at (significant) changes in modularity, rich club coefficient or participation coefficient for the interaction of music and drug specifically? 

Thank you so much for all your help.

Sincerely,

David

Hi David, everything looks correct.

The differences between the two contrasts is that one will test whether connectivity in increased in the music/drug condition, while the other contrast will test for a decrease in connectivity in this condition. In other words, each contrast corresponds to a one-sided alternative hypothesis.

Alternatively, you can select F-test, in which case a two sided alternative hypothesis will be considered (increase or decrease). In this case, you only need to consider the contrast with a "1". 

If you want to test for differences in modularity, rich club etc, you might want to consider the Brain Connectivity Toolbox (BCT). This requires some matlab expertise (BCT does not have a gui). 

Andrew
Originally posted by David de Wide:

Dear Andrew,

Great, I suspected that was the difference based on the areas, but the -1 contrast of Music was very similar to the 1 contrast of the interaction effect (same ROIs, similar t-scores), so I wanted to make sure.

Regarding the BCT, I've already been working with it and made several loop based scripts (for different thresholds) to generate binary matrices and run several of weighted unidirect/binary undirected metrics.

However, with regards to statistical testing, I've only found a way to compare two conditions (i.e. Drug v Placebo or Music v No music), but have been unable to determine how/if to test for an interaction effect using all four conditions. Unlike regular mean scores, I can't subtract the means of the music conditions and then compare those results between drug and placebo. 

Sincerely,

David

Hi David,

it is possible for the ROIs to be the same for the 1 and -1 contrasts, but the actual edges (connections) should generally be different. You might want to try different thresholds. 

Unfortunately, the BCT does not include extensive functionality for statistical testing. You might need to import the network measures computed by BCT into a statistical package that you feel comfortable with and perform hypothesis testing there.

Andrew


Originally posted by David de Wide:

Hey Andrew,

Thank you again for all your help. The NBS is a great tool and has been very useful so far. I've been working hard at applying the NBS and the BCT over the past week and have made quite a bit of progress. However, I do have some remaining quesstion that I haven't been able to find an answer to.


1) After reading several of your articles using the NBS and BCT, I decided to apply partial correlation to my data (similar to the ADHD study) instead of full correlation (similar to the schizoprenia study). However, this changed many of the implicated ROIs dramatically. I am considering presenting both, but am unsure of the merit of partial correlation in this case, as the results are now less intuitive. Which form of correlation do you think is most appropriate? (I also looked into Sparse Partial Correlation, but could not find a matlab plugin).

2) Using the NBS with all 48 scans, and looking at the contrast of the drug effect, I am effectively using both the music and non-music scan together (which themselves are different). Would it be "cleaner" to load a seperate design and dataset of 24 scans and look at the effect of drug seperately for both music and no-music? Or is there an additional contrast I can put in to control for the effect of music on drug (and vice versa)?

3) Before the NBS, I was only aware of regular methods of FDR correction. Using the NBS I initially found significant edges using FDR, despite not finding any when using traditional FDR methods on my 90x90 ANOVA table. I later read about the high number of permutations required, and although this removed all FDR finding, 50.000 is taking a very long time. Is there a lower number that would still be considered "safe"? How do I refer to this method of FDR in my article (i.e. permutation based FDR)?

4) Is there an overview somewhere of which BCT metrics require values to be postive/unsigned?

5) Using Graphvar/GTG, I've been able to do some statistical testing on the network metrics. However, these only allow for a single factor repeated measures, so I still can't look at the contrast of a 2x2 design. Would it be a good to compare each pair seperately and plot all 4 in the same graph?

Thank you for your willingness to answer questions. It's been a massive help, as neither of my supervisors (nor anyone on my floor) has any experience using graph theory based metrics.

Sincerely,

David

Hi David,

1. The choice between full, partial and sparse correlation is difficult. Pros and cons associated with each method. I recommend full correlation in the first instance, but some people in the field would disagree.

2. If you are simply interested in the effect of drug versus no drug, you should delete the music column and the interaction column and rerun the model. You will still need a column for each subject to model each subject's mean. I don't think you should throw away the music scans for this analysis.


3. No. You MUST run at least 50000 - 100000 permutations with FDR. The results will be highly unreliable otherwise. Refer to this as FDR with p-values computed using permutation testing.

4. Generally, the name of the function provides clues about whether the measure is suitable for signed networks. For example, "bu" at the end of the function name means "binary undirected". See BCT webpage for more details. 

5. Your design is repeated in both factors. Therefore a 2 x 2 with only one repeated measure is not appropriate. You could import your data into SPSS and perform testing there.

Andrew

Originally posted by David de Wide:

Hey Andrew,

I've been working hard over the past week on both the NBS and the BCT. I have a few final questions that I hope you would be able to help me with. You've been extremely gratious and  forhtcoming with your expertise, and I honestly could not appericiate it more. I tried getting most of these answers from the available pages of your recent book, but as an unpaid intern living abroad I can't afford it at the moment.

1) So if I understand you correctly, only the functions that include "signed networks" as an option can deal with negative values? (i.e. local_assortativity_wu_sign.m or clustering_coef_wu_sign.m). For the sake of simplicitiy, I feel it is probably wise to stick to binary undirected values. I've given up on all but the original BCT and am just writing scripts instead of a UI. In this case, should I apply some form of normalization on the thresholded matrices before computing/plotting the metrics? 

2) I've seen different approaches to this in different articles, but should I pick a single or small range of thresholds (i.e. 30% or 20-30%, as these values are more often significant than the 30-50% range), or plot changes from density 1 to 50 (relative)? 

3) Follow-up question 2. Should I enter the individual values (1 per subjects so 12 per theshold) into a repeated measures anova with 2x2(x50 in case of 50 density levels), or should I run paired t-tests on each of the pairs (resulting in 50 t-values, one for each density level) and create contrasts by subtracting the measures for music from the no-music scan for each drug, and then a paired t-test between the 2 contrasts? Then conduct FDR on the pvalues?

4) Do I need to control for multiple comparisons for the NBS? Since the effects of music on functional connectivity seem to be diametrically opposed for  (Drug reduces FCD in certain ROIs, which is increased by music, while placebo has higher FCD but is decreased by music) I've looked at both main effects seperately, as well as in the full model. This leaves me with 6 outcomes for the full model (Drug connectivity increase (+) and decrease (-), Music + and -, Interaction + and -), and 8 additional outcomes (Drug effect no-music + and -, Drug effect music + and -, Music effect placebo + and -, and Music effect drug + and -). For each of these outcomes (8 out of 14 have significant edges) I also ran several thesholds from ~3 to ~4.5 to funnel down the number of edges to the most influential. 

5) How should I interpret negative interaction effects? Based on my design matrix(PCB NM, PCB M, Drug NM, Drug M), is the contrast DrugvPCB or PCBvDrug and similar for Music V No music? When calculating t-tests based on mean functional connectivity (mean for each colmun in the 90x90 matrix), I subtracted Music from No-Music and Drug from Placebo. I'm unsure if the NBS uses the same logic in the contrasts. 

6) Most articles seem to only mention the benefit of partial correlation (models direct connection by reducing suprious/unrelated connectivity between the remaining pairs), but what exactly is the downside of this? I've decided to follow your advice and stick to full correlation, but not entirely sure how to defend that decision.

7) Are the p-value for the network and t the t-values for the significant edges saved anywhere in the NBS.out output file? I've saved the NBS file and binary matrix for each analysis but I can only find the alpha level I selected in the UI, but not the lowest p-value that was found for the network). I'm afraid I'll have to run the analyses again and write down each p-value as well as the t-values between the signfiicant edges.

Hopefully these will be my final questions. If you're interested in the results, I can probably send you my internship report once it is finished. Thanks again for your help.

Kind regards,

David

Hi David,
many questions here so I will keep my response brief. You might want to consider visiting a lab/collaborator with expertise in graph analysis.

1. Yes - "signed" means "can deal with negative edge weights". Many different methods to normalize network metrics (e.g. with respect to a degree-matched random graph).

2. Perhaps 5% - 40%. You need to justify what is a reasonable range for your data. You need to decide and carefully justify based on data.

3. Best not to run a test at every threshold. Consider computing the area under the curve (AUC) and perform only on 2 x 2 anova on the AUC which is essentially a summary measure across all thresholds. Take a look at previous graph papers using AUC.

4. I assume you mean multiple comparisons across the different contrasts. Technically you should control for multiple comparisons across independent contrasts but many don't bother.

5. Interactions in NBS are interpreted EXACTLY the same as a typical ANOVA interaction.

6. Partial correlation can underestimate true clustering of network.

7. Yes - all these variables are all stored in the output matrix. Take a look at the reference manual for details.

Andrew
Originally posted by David de Wide:

Dear Andrew
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups (controls -n=36; patients - n=57) who had two randomised drug visits (placebo; drug). Should I simply follow, and generalise, the information you provided to David? With thanks, Liam.

Yes - the advice still holds.


Originally posted by Liam Nestor:

Just to clarify - can I have both groups in the same analysis even though one has 36 subjects and the other has 57? It's not going to matter with the exchange blocks? Best wishes, Liam.

A mismatch in the sample size should not be a problem.


Originally posted by Liam Nestor:

Thanks Andrew
I am following the advice previously given. It's getting everything right, except it's telling me I don't have any exchange blocks, even though I loading an exchange blocks text file (attached) - am I populating the text file properly? Best, Liam.

Hi Liam,

the exchange block file seems to look ok. Perhaps try directly pasting the numbers into the GUI. Note that the software should run without a exchange blocks in any case.

Andrew



Originally posted by Liam Nestor:

Thanks Andrew – pasting them in with a semicolon between each numbers works for the exchange blocks. Best wishes, Liam.

Dear Andrew
    I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups in patients with migraine（Group A：n=29; Group B：n=25）who received two therapy method,separately.(Group A: received real vagus nerve stimulation; Group B received sham Vagus nerve stimulation). so I had four groups(Group A1:before real stimulation,Group A2: after real stimulation, Group B1: before sham stimulation,Group B2: after sham stimulation ).
    I am interested in the main effects of stimulation and time,but more specifically in the interaction effect between stimulation and time. My main issue has been the design matrix and the contrast. Can you help me? Any help would be greatly appericiated.

Sincerely,
zhangyue

I have a similar case in which a participant got once a placebo and once a drug and then did two tasks [high and low]. The explanation here was very helpful in creating a design matrix, so thank you very much! My study has 32 participants and since every participant has 4 matrices, I created a 128x34 matrix. The first column codes drug, the second highVSlow and the rest indicate to which participants they belong. So the first 4 files belonging to subject1 would look like this: 

Sub1PH   0 1 1 0 0 ... 
Sub1PL    0 0 1 0 0 ...
Sub1DH   1 1 1 0 0 ...
Sub1DL    1 0 1 0 0 ... 

The permutation blocks would then look like this [1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, ...]

is this correct? (also see attachment) 

However, 3 things are unclear cause they seem to contradict other posts in this forum:

1. In other posts a column with only ones is additionally included to indicate a global mean. Why is this column here omitted? 
   
2. Similar to the post above, I encoded drug: 0 for placebo and 1 for drug. Why do other posts recommend 1 and -1? I guess there is a difference but I could not figure it out...
   
3. When I am interested in the contrast for 
   

main effect drug = [1 0 0 0 ..]  and for the decrease [-1 0 0 0]
main effect task = [0 1 0 0 .. ] (I guess this means high > low right?) 

I would run a t-test, right? And since I explore both drug > placebo [1 0 0 0 ..]  and placebo > drug [-1 0 0 0], I would need to half the significance to 0.025 to compensate for testing twice, right? Or would I actually need to divide alpha by 4 since I run both contrasts for both main effects to safeguard against alpha error accumulation? 

Thank you so much for your help! 

Cheers, 

Max 

 

 

 

Hi Max,

the design matrix looks ok. It seems that an additional 1 is missing in the permutation blocks. 

1. A global mean should not be included. This is because you are modelling a separate mean for each subject. 
   
2. Whether you code with -1/1 or 0/1 won't have any impact on the p-values. Both options are fine. 
   
3. In terms of correcting for multiuple comparisons, this would depend on the nature of your hypothesis. The strictest option would be to divde by 4. But that may be too conservative. 
   

You may also wanted to test for an interaction between task and drug condition. This would probably be the most interesting contrast. 

Andrew

Originally posted by Max Kathofer :

I have a similar case in which a participant got once a placebo and once a drug and then did two tasks [high and low]. The explanation here was very helpful in creating a design matrix, so thank you very much! My study has 32 participants and since every participant has 4 matrices, I created a 128x34 matrix. The first column codes drug, the second highVSlow and the rest indicate to which participants they belong. So the first 4 files belonging to subject1 would look like this: 

Sub1PH   0 1 1 0 0 ... 
Sub1PL    0 0 1 0 0 ...
Sub1DH   1 1 1 0 0 ...
Sub1DL    1 0 1 0 0 ... 

The permutation blocks would then look like this [1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, ...]

is this correct? (also see attachment) 

However, 3 things are unclear cause they seem to contradict other posts in this forum:

1. In other posts a column with only ones is additionally included to indicate a global mean. Why is this column here omitted? 
   
2. Similar to the post above, I encoded drug: 0 for placebo and 1 for drug. Why do other posts recommend 1 and -1? I guess there is a difference but I could not figure it out...
   
3. When I am interested in the contrast for 
   

main effect drug = [1 0 0 0 ..]  and for the decrease [-1 0 0 0]
main effect task = [0 1 0 0 .. ] (I guess this means high > low right?) 

I would run a t-test, right? And since I explore both drug > placebo [1 0 0 0 ..]  and placebo > drug [-1 0 0 0], I would need to half the significance to 0.025 to compensate for testing twice, right? Or would I actually need to divide alpha by 4 since I run both contrasts for both main effects to safeguard against alpha error accumulation? 

Thank you so much for your help! 

Cheers, 

Max 

 

 

 

 

Dear Andrew,

 

Thanks for the quick response!

1. Thanks for catching the missing 1 in the permutation vector (was a copy error on my part)
   
2. If drug contrast tests [0 1 0 0 ..] or [0 -1 0 0 ..]. the interpretation is: which network cluster is increased during drug over placebo and the other one would be which network is more active during placebo over drug right? But they do not need to yield symmetrical effects right? So if I find a cluster for contrast 1, I do not need to find a cluster for -1, right?
   
3. The interaction between drug and high low would be coded like this I guess: 
   

Sub1PH   0 1 0 1 0 0 ... 
Sub1PL    0 0 0 1 0 0 ...
Sub1DH   1 1 1 1 0 0 ...
Sub1DL    1 0 0 1  0 0 ... 

(First column main effect drug, second main effect highvlow, third column interaction drugxHigh) 

The interpretation of this effect would be: is there a network that is more activated during drug and high conditions compared to everything else, right? [contrast: [0, 0, 1, 0 , 0 ..]

 

Last question, does the interaction reduce the power to detect main effects? E.g. normally using behavioral data I  would test which regressors to include and then use the model that performs best (BIC or AIC). I usually do this since interaction terms use a lot of df and thus - if not beneficial for the glm - can result in decreased power to detect other effects right? Is there a potential with NBS in this regard as well? Can I test e.g. wether including this effect is necessary?  

 

Thanks so much! 

Cheers, 

Max 

 

2. Yes, assuming 1 is used to indicate the drug condition and 0 for the other condition. They effects do not need to be symmetric. 

3. Yes - the interaction looks correct. For an interaction effect, using 1/-1 rather than 0/1 can assist with interpretation, but the p-values will be the same in both cases. 

Yes - adding an interaction could reduce the ability to detect main effects (it will also increase the number of multiple comparisons), although this characteristic is not specific to the NBS and the same would hold true for any kind of regression model. 

 

 

Originally posted by Max Kathofer :

Dear Andrew,

 

Thanks for the quick response!

1. Thanks for catching the missing 1 in the permutation vector (was a copy error on my part)
   
2. If drug contrast tests [0 1 0 0 ..] or [0 -1 0 0 ..]. the interpretation is: which network cluster is increased during drug over placebo and the other one would be which network is more active during placebo over drug right? But they do not need to yield symmetrical effects right? So if I find a cluster for contrast 1, I do not need to find a cluster for -1, right?
   
3. The interaction between drug and high low would be coded like this I guess: 
   

Sub1PH   0 1 0 1 0 0 ... 
Sub1PL    0 0 0 1 0 0 ...
Sub1DH   1 1 1 1 0 0 ...
Sub1DL    1 0 0 1  0 0 ... 

(First column main effect drug, second main effect highvlow, third column interaction drugxHigh) 

The interpretation of this effect would be: is there a network that is more activated during drug and high conditions compared to everything else, right? [contrast: [0, 0, 1, 0 , 0 ..]

 

Last question, does the interaction reduce the power to detect main effects? E.g. normally using behavioral data I  would test which regressors to include and then use the model that performs best (BIC or AIC). I usually do this since interaction terms use a lot of df and thus - if not beneficial for the glm - can result in decreased power to detect other effects right? Is there a potential with NBS in this regard as well? Can I test e.g. wether including this effect is necessary?  

 

Thanks so much! 

Cheers, 

Max 

 

 

Hi Andrew! 

 

Thanks for all your responses! I just have one last question. What if I want to model a continuous predictor and check whether there exists a linearly increasing subgraph? 

Again I have a repeated measure design across 2 drug conditions but I have ~20 FC matrices for each drug condition per subject. I would have 2 questions:

1. It is often recommended to demean data across ALL participants. I am unsure why though. Wouldn't it make more sense to demean for each subject across drug conditions to account for their individual differences?
   
2. Would a design matrix like this then be correct? 
   

 

0 -2 1 0 0 0 ....

0 3 1 0 0 0 .... 

1 -0.5 1 0 0 0 ....

0 -8 0 1 0  0 0 

 

The first column denotes drug (0/ placebo; 1/drug) 
The second column is the per subject demeaned behavioral rating
Column 3 denotes that the first 3 entries belong to participant 1 
Column 4 denotes that the 4th entry belongs to subject 2 
The zeros afterwards would have length(subejct) 

 

Thank you for your help! 

Cheers, 

Max 

Hi Max,

Demeaning can be helpful when studying an interaction effect. If you are looking at the interaction of two main effects, A and B, demeaning A and B ensures that coefficient for A reflects the mean value for this variable in the interaction effect (instead of when the variable is 0, which might not be meaningful). In this way, demeaning can help with interpretting the model, but it won't change statistical signficiance. 

The design matrix looks correct. You may also want to study the interaction between group and behaviour rating. This is acheived by multiplying the group and behaviour column. In this case, the group column would need to be removed in the final model.

Best, 

Andrew

 

 

Originally posted by Max Kathofer :

Hi Andrew! 

 

Thanks for all your responses! I just have one last question. What if I want to model a continuous predictor and check whether there exists a linearly increasing subgraph? 

Again I have a repeated measure design across 2 drug conditions but I have ~20 FC matrices for each drug condition per subject. I would have 2 questions:

1. It is often recommended to demean data across ALL participants. I am unsure why though. Wouldn't it make more sense to demean for each subject across drug conditions to account for their individual differences?
   
2. Would a design matrix like this then be correct? 
   

 

0 -2 1 0 0 0 ....

0 3 1 0 0 0 .... 

1 -0.5 1 0 0 0 ....

0 -8 0 1 0  0 0 

 

The first column denotes drug (0/ placebo; 1/drug) 
The second column is the per subject demeaned behavioral rating
Column 3 denotes that the first 3 entries belong to participant 1 
Column 4 denotes that the 4th entry belongs to subject 2 
The zeros afterwards would have length(subejct) 

 

Thank you for your help! 

Cheers, 

Max 

 

Thank you so much for the answer! 

So given I want to investigate an interaction effect, I would need to remove the drug condition afterwards? 

 

Meaning a design matrix like this:

Drug | Rating | Participant columns ...

0 3.2 1 0..

0 -1.8 1 0..

1 1.2 1 0...

1 -9 1 0...

1 -4 0 1... 

first 4 rows denote first subject, 5th row 2nd subject 

would become 

Interaction | Participant 

0 1 0...

0 1 0..

1.2 1 0...

-9 1 0.. 

-4 0 1 

 

Thank you so much!