I have precisely the same attitude - not just to the phone either, but to things like Messenger. I refuse to be at everyone's beck and call, and will reply when it is convenient to do so. You'd be amazed how many people get offended by such an attitude, but it's no skin off my nose, and I couldn't care less if they are.