# Look at my mouse (3D Screen to World)

I need a little help. I’ll be glad if anyone can show me the right direction…

My problem is actually quite simple; follow the mouse position on 3D even when camera is rotated.

So I use screen to world calculation. It works great until I start to rotate the camera around and I’m not sure how to handle this situation cause I thought projection matrix is enough to handle and I see it is not(I might be wrong of course)

My screen to world is similar to Rendy’s, except I’m not using frustum.
There is no custom render.script, just rotating the camera using Orbit Camera

Minimal project is here: GitHub - selimanac/look-at-my-mouse

``````local function screen_to_world(self)
local inv = vmath.inv(self.camera_projection * self.camera_view)

-- Center
local center_x = self.mouse_screen_position.x - ((DISPLAY_WIDTH / 2))
local center_y = self.mouse_screen_position.y - ((DISPLAY_HEIGHT / 2))

-- Near & Far plane
local near_world_position = vmath.vector4(inv * vmath.vector4(center_x, center_y, -1, 1))
local far_world_position = vmath.vector4(inv * vmath.vector4(center_x, center_y, 1, 1))

near_world_position = near_world_position / near_world_position.w
far_world_position = far_world_position / far_world_position.w

local frustum_z = (self.mouse_screen_position.z - self.camera_nearZ) / (self.camera_farZ - self.camera_nearZ)
local world_position = vmath.lerp(frustum_z, near_world_position, far_world_position)

self.mouse_world_position.x = world_position.x
self.mouse_world_position.y = world_position.y
self.mouse_world_position.z = world_position.z
end

function update(self, dt)
-- ROTATE TO MOUSE
if self.mouse_screen_position then
self.camera_projection = go.get(camera_id, "projection")
self.camera_view = go.get(camera_id, "view")
b_screen_to_world(self)
end

if self.mouse_world_position then
self.hero_rotation_angle = math.atan2((self.mouse_world_position.x - self.hero_position.x), (self.mouse_world_position.y - self.hero_position.y)) - math.pi
self.hero_rotation = vmath.quat_rotation_y(-self.hero_rotation_angle)
go.set_rotation(self.hero_rotation, hero_id)
end
end
``````

So, I know this is not a easy task to ask and this is just a hobby project, but I’ll be glad to hear your suggestions.

Not the solution but random thoughts:

1. rotate the player while rotating the camera so you won’t “lose” the mouse cursor
2. at the end of the camera rotation call a function to rotate the player to face the cursor.
1 Like

I don’t know how it works with these extensions, but with the old and well made Rendercam it’s simple:

``````local up 	= vmath.vector3(0, 1, 0)
local pivot = vmath.vector3(0, 8, 0)
...
self.look_at = rendercam.screen_to_world_plane(action.screen_x, action.screen_y, up, pivot)
...
``````

works with any camera angle because it doesn’t matter. We know the position of objects and the cursor in the world coordinate system.

Also I prefer `euler.y` instead quat for the hero rotation.

``````local angle = lume.angle(self.look_at.x, self.look_at.z, self.position.x, self.position.z)
self.angle = -math.deg(angle) - 90 -- depends on origin model rotation
local old_angle = go.get(self.dude, "euler.y")
go.set(self.dude, "euler.y", math3d.lerp_angle(.92, self.angle, old_angle))
``````
1 Like

Actually I’m not using any extensions, I don’t want to.
Orbitcam is just a basic default camera rotation script which I don’t want to spend time on it.
Rendercam is a good old big boy I’ll check `screen_to_world_plane` source. But I don’t want to touch the default(and new) render.script(no reason, just a challenge).

Any reason to prefer it? (easy, accurate…)

Thank you

1 Like

Might help, I’ll try.

Thank you

Hmm… The first problem is you need to convert the screen position into -1 to +1 coordinates. In other words: you should be dividing by the window size in there somewhere.

Next, what is the “screen” Z position you are using? The way you are using it in the code, it’s a distance from the camera in the view direction, but what you seem to want for your actual game is a point on a 3D plane, so those two things don’t match up.

Yes, I manage to make `screen_to_world_plane` work with build-in camera and without modify the render.script. It is better now. But, I would like to try to fix my initial code by your suggestions (for learning purposes)