I can completely understand your perspective. I think there are a variety of factors that determine how people respond to this film, including personality, background, temperament, and so forth. I don't think it is possible for anyone to be a neutral spectator, not fully in any case, although I will concede that I sit very much at the extreme end of that spectrum. I always try to approach a film analytically, especially if it is a contentious work such as this one.
I must also add that whilst it is true that gender can be a determining factor in terms of response, I know several women who take a similarly analytical approach when critiquing this film. I also know several others who actively enjoyed the film or consider it a favourite (check out the rest of the comments for evidence of this). Conversely, you'll find there are men who find it profoundly disturbing and can't get past that either (again, see the comments here for evidence).
What I think all this illustrates is that 1) In this case, the viewer is the final piece of the puzzle that determines the meaning of the film (even if, as in my case, I still can't quite decide what that is), 2) that said interpretations are going to wildly vary, as we are all very different people, and 3) that as you point out, Stanley Kubrick was a genius.
I'd just like to add that I really appreciate you reading my pieces, and your comments are always insightful and interesting. Thank you! :)