I found [url=https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=FSL;18112e43.0903]a discussion with Thomas Nicols on this issue,[/url] it seems my reasoning was quite close to what he says there, but apparently he affirms this is not an issue (except that the test might not be very sensitive).

In other words, applying a permutation test on a single-subject against group study is only risking being too conservative. Which is very fine!