Well, I don't think swinging fists really resolves anything. I'm sure there are one or two scenarios, as you say, but I'm pleased to tell you I've not encountered them.
The last time I was in a proper fight was at the age of 16 in a ridiculous drunken pub brawl over something utterly trivial (I was acting in self defence - even then, I wasn't stupid enough to start fights).
Regarding Connery, I suppose I've just always had a healthy compartmentalisation of actor versus character, even as a child. Probably a personality thing. :)