help
help > RE: Results inconsistent and permutation number
May 28, 2014 06:05 AM | Andrew Zalesky
RE: Results inconsistent and permutation number
Dear Hank,
Two suggestions:
1. I suggest using a much larger number of permutations when using the "FDR option". Perhaps 100,000 or more. This might require a lot of time to compute. When using a sufficient number of permutations, the results will stabilize across different trials. This is because a p-value for each link is computed non-parametrically. In future releases, we will compute p-values for the FDR option parametrically.
2. You might also want to try the "NBS option", which will require substantially fewer permutations (i.e. 5,000).
Andrew
Originally posted by Sung-han Lin:
Two suggestions:
1. I suggest using a much larger number of permutations when using the "FDR option". Perhaps 100,000 or more. This might require a lot of time to compute. When using a sufficient number of permutations, the results will stabilize across different trials. This is because a p-value for each link is computed non-parametrically. In future releases, we will compute p-values for the FDR option parametrically.
2. You might also want to try the "NBS option", which will require substantially fewer permutations (i.e. 5,000).
Andrew
Originally posted by Sung-han Lin:
Dear Professor Andrew,
Thanks for your network based statistic (NBS) toolbox! Recently I used the NBS toolbox to calculate the structural connectivity differences between groups.
Firstly, I tried the t-test with FDR to find out the linked-based network between two groups. The connectivity matrix is 90*90, permutation number was set as 10,000 as recommended.
While I clicked the FDR button, the toolbox gave a results (Results_1) after calculation. Then I tried to increased the permutation number to 15,000 and I got another result (Results_2). Finally, I decreased the permeation number back to 10,000 and the finally result came out (Results_3).
I assume the Results_1 and Results_3 would be the same because of the same setting and Results_2 would find fewer nodes and edges.
However, the Results_3 is not the same with Results_1 but the same with Results_2. Why did I get these results? Did I do anything wrong with this procedure or should I set anything else?
The other question is about the permutation number. Is the number of 10,000 sufficient or not? or the number should be varied with my sample size and matrix size?
Thanks for your help!
Sung-han Lin (Hank)
Thanks for your network based statistic (NBS) toolbox! Recently I used the NBS toolbox to calculate the structural connectivity differences between groups.
Firstly, I tried the t-test with FDR to find out the linked-based network between two groups. The connectivity matrix is 90*90, permutation number was set as 10,000 as recommended.
While I clicked the FDR button, the toolbox gave a results (Results_1) after calculation. Then I tried to increased the permutation number to 15,000 and I got another result (Results_2). Finally, I decreased the permeation number back to 10,000 and the finally result came out (Results_3).
I assume the Results_1 and Results_3 would be the same because of the same setting and Results_2 would find fewer nodes and edges.
However, the Results_3 is not the same with Results_1 but the same with Results_2. Why did I get these results? Did I do anything wrong with this procedure or should I set anything else?
The other question is about the permutation number. Is the number of 10,000 sufficient or not? or the number should be varied with my sample size and matrix size?
Thanks for your help!
Sung-han Lin (Hank)
Threaded View
| Title | Author | Date |
|---|---|---|
| Sung-han Lin | May 28, 2014 | |
| Andrew Zalesky | May 28, 2014 | |
| Sung-han Lin | May 30, 2014 | |