A moral imperative (cf. Kant) is supposed to be universal. If you believe in free speech, you defend it even when it protects speech you hate, not only when it benefits your side.