I think there are ways of doing it so it's obvious you meant to, and why. It can be very interesting, in a story, to go from one head into the other and see things a different way; it can also be quite moving, as you realise what character A is saying comes acrosso completely wrong to character B, or whatever.

Accidental slips can be awful, but as usual, ain't no rule in writing that someone can't break to good effect.