You are free to take a range of stances. If I disagree with you (or you with me) it doesn’t necessarily mean that I am against you (or you against me).
This is SO basic, I wonder how some people can be so easily misled to think this ‘with us or against us’ statement is at all reasonable. People who think like this, think like extremists – because, by definition, you are restricting choices to two opposing ‘extremes’. Lighten-up. Be more real.