help
help > contrast ANOVA
Nov 10, 2017 12:11 AM | Andrew Zalesky
contrast ANOVA
Hi Charnaya,
The two design that you have provided are equivalent.
For FDR, all results that use only 5000 permutations are unreliable and should not be used. For FDR, at least 50000 to 100000 permutations are required. This is made clear in the manual.
FDR requires more permutations because the permutations for FDR are used to compute a p-value for each edge, whereas the NBS uses permutation to compute a p-value for components. For FDR procedures such as Benjamini-Hochberg, we need accurate p-values out to several decimal places, and this requires many more permutations to achieve. For NBS, we generally only report p-values to a few decimal places (e.g. p<0.05).
In future releases, the current FDR implementation will be replaced with parametric FDR.
Andrew
Originally posted by Charanya Muralidharan:
The two design that you have provided are equivalent.
For FDR, all results that use only 5000 permutations are unreliable and should not be used. For FDR, at least 50000 to 100000 permutations are required. This is made clear in the manual.
FDR requires more permutations because the permutations for FDR are used to compute a p-value for each edge, whereas the NBS uses permutation to compute a p-value for components. For FDR procedures such as Benjamini-Hochberg, we need accurate p-values out to several decimal places, and this requires many more permutations to achieve. For NBS, we generally only report p-values to a few decimal places (e.g. p<0.05).
In future releases, the current FDR implementation will be replaced with parametric FDR.
Andrew
Originally posted by Charanya Muralidharan:
Hello Dr. Zalensky,
We are trying to perform ANOVA with our test sample set of 52 individuals (4 independent groups). I have a question about design matrix, contrast and FDR method. We followed the suggestion provided by you in this forum with reference to design matrix and contrast settings. We tested the following design and contrast settings.
Design 1:
1 1 0 0
1 1 0 0
1 -1 1 0
1 -1 1 0
1 0 -1 1
1 0 -1 1
1 0 0 -1
1 0 0 -1 Contrast set to [0 1 1 1]
Design 2:
1 0 0 0
1 0 0 0
0 1 0 0
0 1 0 0
0 0 1 0
0 0 1 0
0 0 0 1
0 0 0 1 Contrast set to [1 1 1 1]
I ran both these settings with F-test and FDR set to permutations 50000. Both of them resulted in no significant results. However, when I tested them with permutation of 5000, design 1 gave me more significant edges than design 2. Considering both the matrices and their corresponding contrasts are equivalent, why would one see a discrepancy? Is it due to insufficient number of permutations? Is there a resource explaining why FDR requires more permutations than NBS?
Your thoughts would be really helpful!
Thank you
Charanya
We are trying to perform ANOVA with our test sample set of 52 individuals (4 independent groups). I have a question about design matrix, contrast and FDR method. We followed the suggestion provided by you in this forum with reference to design matrix and contrast settings. We tested the following design and contrast settings.
Design 1:
1 1 0 0
1 1 0 0
1 -1 1 0
1 -1 1 0
1 0 -1 1
1 0 -1 1
1 0 0 -1
1 0 0 -1 Contrast set to [0 1 1 1]
Design 2:
1 0 0 0
1 0 0 0
0 1 0 0
0 1 0 0
0 0 1 0
0 0 1 0
0 0 0 1
0 0 0 1 Contrast set to [1 1 1 1]
I ran both these settings with F-test and FDR set to permutations 50000. Both of them resulted in no significant results. However, when I tested them with permutation of 5000, design 1 gave me more significant edges than design 2. Considering both the matrices and their corresponding contrasts are equivalent, why would one see a discrepancy? Is it due to insufficient number of permutations? Is there a resource explaining why FDR requires more permutations than NBS?
Your thoughts would be really helpful!
Thank you
Charanya
Threaded View
| Title | Author | Date |
|---|---|---|
| Andreas Hahn | Nov 14, 2014 | |
| Andrew Zalesky | Nov 16, 2014 | |
| Charanya Muralidharan | Nov 9, 2017 | |
| Andrew Zalesky | Nov 10, 2017 | |
| Charanya Muralidharan | Nov 13, 2017 | |
| Yuan-Fang Zhao | Oct 14, 2016 | |
| Andrew Zalesky | Oct 15, 2016 | |
| Andreas Hahn | Nov 18, 2014 | |
| Andrew Zalesky | Nov 19, 2014 | |