Wednesday, October 28, 2009

How to select 3D object from cursor/touch in OpenGL ES

When writing 3D games there is a requirement quite common (forget about OpenGL ES at the moment):

How to select an 3D object in the scene, by mouse/touch?

This problem is not so trivial as it seems. Actually the more you think about it, the more you will find it is not so easy to solve.

Basically, we are trying to map the windows coordinate back to the 3D coordinate. If you understand how an object in OpenGL coordinate is projected onto your 2D display:

screen_vector = ((3D_vector_4x1) * model_view_matrix_4x4) * projection_matrix_4x4;

All we need to do is do the reverse, multiply the screen_vector with the reverse of model view/projection matrices:

3D_vector_4x1 = (screen_vector * (projection_matrix_4x4_inv)) * model_view_matrix_4x4_inv;

Actually, in desktop version of OpenGL, you don't have to do the calculation yourself. There is the mighty gluUnproject that handles it for you.

But there is still a problem. Since your LCD screen is 2D, you only get the x,y component of screen_vector. Where is z? z is actually the depth component and you can find it in the depth buffer. In desktop version of OpenGL, you can call glReadPixels with GL_DEPTH_COMPONENT flag. Of course it is not the full story: the depth buffer only stores pixel depth values that are minimal (unless you change the OpenGL depth function), when transformed back to 3D_vector_4x1 you only get the nearest object. Usually it is desired, but if your wanted object is painted behind other transparent polygons, there is a little bit of problem - but hey it's not a big deal.

But when comes to OpenGL ES, there is a big problem implementing all these:
1. You don't get GLU package. So there is no gluUnproject.
2. You don't have access to the depth buffer. (No GL_DEPTH_COMPONENT for you!)

So the previous solution won't work!
Face it, in OpenGL ES it is a one-way road: you can get windows coordinate from a 3D coordinate, but there is no other way around. Then how to solve the problem?

Well... brutal force always works. First, you define what object is selectable in the game world. Second, you calculate the window coordinate of that object during OpenGL paint, keeping it updated. Last, when a mouse click/touch happens, compare the event windows coordinate to all existing window coordinates that updated during OpenGL paint, and pick one that is nearest. Of course, it is not universal and not all objects in the world can be selected; but hey, do you really need that?

Or you can use ray-tracing.

No comments:

Post a Comment