Yeah, I bet that’s your problem then. You player body can contact those inside edges and give you a sideways normal.
Honestly I don’t think I’ve ever found a really great solution for this. You can try making your player’s collision rounded or beveled on the bottom. You can try to make checks to ignore these contacts, with some sort of threshold, or have vertical contacts override horizontal ones . . .
Actually I think the only reliable way to deal with this is to rely on raycasts instead of contact point responses.