Yes, you need to configure your atlas. Select the root of the atlas in the Outline and change Extrude Borders from 0 to 1 pixels. What you’re seeing is something called texture bleeding and it’s usually resolved by extruding/repeating the border pixels of each individual image in the atlas 1 or more times.
This is mentioned in the manual and in multiple places on the forum, for instance in this post.
Please continue to ask questions! We’re here to help! We should probably improve the documentation surrounding the whole atlas configuration and visual artefacts. This is a question that keeps popping up here on the forum (ping @sicher).