Transform screen pos to world pos (and back) with perspective camera (DEF-848)

Feature requests
#1

I’m using a perspective camera and I want to be able to convert to and from screen coordinates and world coordinates (at z = 0). I don’t really care about rotation but I need to deal with camera movement in all directions and window resizing. I’ve tried to muddle through it in various ways, but nothing that I thought made sense gave me a meaningful output. I guess I’m searching for the wrong words because I haven’t been able to find anything useful about this, either on this forum or the internet at large.

I’ve done it with an orthographic camera just by adding in the camera offset and multiplying by a uniform zoom value used in the render script, but for some projects I want to use a 3D perspective camera.

Thanks for any help.

#2

There is no built in support for this currently, but we have an issue in the backlog to redesign the camera component and include some sort of functionality to facilitate screen<->world space conversions. DEF-848

1 Like
#3

You could also do the math yourself in lua until we have support for it: http://stackoverflow.com/questions/7692988/opengl-math-projecting-screen-space-to-world-space-coords

Check the vmath module for help functions.

4 Likes
#4

Yeah, it would be cool to have some built-in functions for this, but not a huge priority. The math help is what I was looking for. Matrix math is not exactly my strong point. I got it working now, thanks a lot!

2 Likes
#5

Great! Please share the code if you want to. Maybe it will benefit someone else as well.

2 Likes
#6

Well . . . I thought I had it working, it turns out it was a bit more complicated. But now I do actually have it working, after some help from my brother, and here’s the example project:

Link removed, see the next post

The tricky part is the z value. In that stack overflow answer, he just says get it with “glReadPixel” or “manually go from -1 to 1 ( zNear, zFar )”. What he neglects to say is that the -1 to 1 z value is not linear at all, it’s some funky curve. So you can’t just figure it out from your camera z pos and near/far settings in a simple way as you would expect. The easiest way is instead to take the points at the cursor x and y on the near and far planes (z= -1 and 1), transform those into world-space with the view*projection matrix, and then lerp between those to the z of your desired world plane (presumably 0, but it can be whatever).

The awesome part is, with this you can move and rotate the camera however you want and change the window size however you want, and it will still give you the correct world-space point under your screen-space cursor with no extra fuss. The example includes both the screen-to-world and the world-to-screen transforms, plus camera zoom and pan.

The only thing I didn’t get working is the world-to-screen transformation with changing window size and the GUI “Adjust Reference” mode enabled (“Per Node”). I’m not sure what’s going on there. If you set “Adjust Reference” to “Disabled” or just don’t resize the window, it works fine.

5 Likes
#7

Update for Forum Searchers:

I’ve completely redone and expanded my old perspective screen_to_world project and neatly packaged it up as “Rendercam”. You can get it from the Community Asset Portal, or Github.

My old screen_to_world project only actually worked in one specific circumstance, and didn’t cover a lot of things. Rendercam actually works for every situation (so far . . . ), has a lot more features, and can be used easily as a library dependency. It can do orthographic cameras as well (just a checkbox!), and the extra world-to-screen calculations for each GUI adjust mode.

5 Likes