I'm also not a hugger and have also often used my wife as a human shield in the past. These days, I don't give a crap what anyone thinks. I flat-out refuse hugs and the ensuing death stare for anyone who crosses into my air space freezes them in their tracks.
In truth, I am much like you when it comes to hugs. Of course I hug my wife and children. But for me, hugs are about trust. Total trust. The only other people I permit to hug me are immediate family members and my best friend. That's it.
As a teenager, I once joked that I had no problem with casual sex, but hugs? Forget it.