It's not just the time, it's the sheer inconvenience and stress of changing one's routine and having to keep it all secret. My wife once said to me that if I decide to have an affair that she'd make the arrangements, as she knows how much changes to my routine wind me up.