Yes, you need to configure your atlas. Select the root of the atlas in the Outline and change Extrude Borders from 0 to 1 pixels. What you’re seeing is something called texture bleeding and it’s usually resolved by extruding/repeating the border pixels of each individual image in the atlas 1 or more times.
This is mentioned in the manual and in multiple places on the forum, for instance in this post.