OpenXR/OpenVR (virtual reality support)

[kwahoo]

This thread continues discussion from a thread about Oculus Rift.
I would like discuss here about wider VR hardware support. A few weeks ago an open VR/AR standard has been was presented: OpenXR.


UPDATE:

I stopped working on the Python script and moved to C++. The code can be found in this fork https://github.com/kwahoo2/FreeCAD
Notes about compilation:
0. Download and install OpenVR SDK from https://github.com/ValveSoftware/openvr
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

Additional notes about compilation on Windows:
1. OpenVR_LIBRARY should point to lib/win64/libopenvr_api.lib distributed with the OpenVR SDK
2. libopenvr_api.dll can be found in Steam\steamapps\common\SteamVR\bin\win64

End of update.

Waiting for OpenXR hardware adoption (manufacturers like Oculus or Valve announced future support) I've started experimenting with another API: OpenVR.
OpenVR is not quite open as name suggest - it still needs a proprietary runtime (SteamVR), but unlike Oculus SDK it is supported on different operating systems and supports hardware from other manufacturers.

Initially, I started tinkering with jriegel Rift's implementation, but later i found Python bindings for OpenVR pyopenvr. Basic idea is: write (and experiment with) an implementation in pyopenvr, and then port the code to C++.

Here is the repository: https://github.com/kwahoo2/freecad-pyopenvr
Steps to run the code:
1. Install pyopenvr, and SDL + OpenGL Python packages
2. Run SteamVR
3. Paste the code into FreeCAD

Famous Schenkel in VR:



Known issues:
1. The code renders two exactly same pictures - wrong buffer configuration?
2. Head tracking is not implemented yet
3. SoFrustumCamera needs further investigation. I'm trying to get necessary variables for the camera from IVRSystem::GetProjectionMatrix and IVRSystem::GetEyeToHeadTransform matrices, but maybe there is a better way? Update: Use IVRSystem::GetProjectionRaw

Future work:
1. Fix issues
2. Remove SDL dependency, use QOpenGLWidget (?) Update: Check the new offscreen rendering driver in SDL https://hg.libsdl.org/SDL/rev/9692ad570003
3. Port to C++ Initial OpenVR C++ port https://github.com/kwahoo2/FreeCAD/comm ... b8ab0c1e28
4. Add basic navigation
5. Add controllers support (for navigation)
6. Move to OpenXR

[Kunda1]

Steps to run the code:
1. Install pyopenvr, and sdl + opengl python packages
2. Run SteamVR
3. Paste the code into FreeCAD

Known issues:
1. The code renders two exactly same pictures - wrong buffer configuration?
2. Head tracking is not implemented yet
3. SoFrustumCamera needs further investigation. I'm trying to get necessary variables for the camera from IVRSystem::GetProjectionMatrix and IVRSystem::GetEyeToHeadTransform matrices, but maybe there is a better way?

Future work:
1. Fix issues
2. Remove SDL dependency, use QOpenGLWidget (?)
3. Port to C++
4. Add basic navigation
5. Add controllers support (for navigation)
6. Move to OpenXR

@kwahoo
Fantastic!
Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily. You can also create a Roadmap etc...

[kwahoo]

Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily.

Done. I updated the original post.

At this moment mostly(at least I hope so...) correct rendering with rotation and position tracking is possible.

I found the culprit of rendering the same image twice:
creating two scenes this way:

Code: Select all

rootScene = [SoSeparator()] * 2
...
for eye in range(2):
  m_sceneManager.setSceneGraph(rootScene[eye])

does not work. The second scene changes the first one.

[schurik]

This sounds great.
Is it possible to run my Rift S on CATIA with this?

[kwahoo]

Is it possible to run my Rift S on CATIA with this?

Rift S: Yes.
Catia: No, at least not with this script.

----
I have some issues to be ironed out, there is probably an error in cameras configuration (objects are too big, moving too fast). Update: this needs more testing, added a simple test file to the repository: a room with pillars and a table.

But meanwhile some performance teaser (click to see a bigger pic):


To be honest, I was expecting worse fps;)

[Kunda1]

Sent another PR to soup up the README.

[kwahoo]

Thanks!

There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.

[Kunda1]

There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.

I would pose this as a question on the Open Issues subforum so it gets more exposure.

[kwahoo]

I ported a basic OpenVR support to C++. It has the same functionality as the Python script.

The implementation can be found in this repository https://github.com/kwahoo2/FreeCAD Commit introducing OpenVR https://github.com/kwahoo2/FreeCAD/comm ... b8ab0c1e28

Notes about compilation:
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

There is no OS-specific code in the implementation - it should run both on Linux and Windows. Mac has some beta SteamVR support, but I do not have any experience with it .

Running a scene:
1. Start SteamVR
2. Open a model in the FreeCAD
3. Run View->FreeCAD-OpenVR