It's usually not a mood breaker for us either. I'll grab a different toy or switch batteries, whichever feels most natural at the moment. But I always keep spare batteries in the toy chest, so they are right there and totally handy. I like to be prepared and have everything within arm's reach for convenience's sake. That way you don't have to get up and rummage through the junk drawer looking for new batteries.
To be totally honest, I'm actually so anal retentive, I now almost always check the batteries before we start. I'll only keep the same batteries in a toy for about 2 sessions before switching them, because I like battery toys at full power. Doing that, we haven't had a battery die mid-session for quite a while. We have had toys break mid-session, but that didn't have anything to do with batteries. Oh the perils of a good sex life! LOL