OpenXR/OpenVR (virtual reality support)

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
kwahoo
Posts: 235
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

OpenXR/OpenVR (virtual reality support)

Postby kwahoo » Sat Sep 21, 2019 7:05 pm

This thread continues discussion from a thread about Oculus Rift.
I would like discuss here about wider VR hardware support. A few weeks ago an open VR/AR standard has been was presented: OpenXR.


UPDATE:

I stopped working on the Python script and moved to C++. The code can be found in this fork https://github.com/kwahoo2/FreeCAD
Notes about compilation:
0. Download and install OpenVR SDK from https://github.com/ValveSoftware/openvr
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

Additional notes about compilation on Windows:
1. OpenVR_LIBRARY should point to lib/win64/libopenvr_api.lib distributed with the OpenVR SDK
2. libopenvr_api.dll can be found in Steam\steamapps\common\SteamVR\bin\win64

End of update.

Waiting for OpenXR hardware adoption (manufacturers like Oculus or Valve announced future support) I've started experimenting with another API: OpenVR.
OpenVR is not quite open as name suggest - it still needs a proprietary runtime (SteamVR), but unlike Oculus SDK it is supported on different operating systems and supports hardware from other manufacturers.

Initially, I started tinkering with jriegel Rift's implementation, but later i found Python bindings for OpenVR pyopenvr. Basic idea is: write (and experiment with) an implementation in pyopenvr, and then port the code to C++.

Here is the repository: https://github.com/kwahoo2/freecad-pyopenvr
Steps to run the code:
1. Install pyopenvr, and SDL + OpenGL Python packages
2. Run SteamVR
3. Paste the code into FreeCAD

Famous Schenkel in VR:
Image


Known issues:
1. The code renders two exactly same pictures - wrong buffer configuration?
2. Head tracking is not implemented yet
3. SoFrustumCamera needs further investigation. I'm trying to get necessary variables for the camera from IVRSystem::GetProjectionMatrix and IVRSystem::GetEyeToHeadTransform matrices, but maybe there is a better way? Update: Use IVRSystem::GetProjectionRaw

Future work:
1. Fix issues
2. Remove SDL dependency, use QOpenGLWidget (?) Update: Check the new offscreen rendering driver in SDL https://hg.libsdl.org/SDL/rev/9692ad570003
3. Port to C++
Initial OpenVR C++ port https://github.com/kwahoo2/FreeCAD/comm ... b8ab0c1e28
4. Add basic navigation
5. Add controllers support (for navigation)
6. Move to OpenXR
Last edited by kwahoo on Tue Nov 19, 2019 7:08 pm, edited 7 times in total.
User avatar
Kunda1
Posts: 5799
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Postby Kunda1 » Sat Sep 21, 2019 7:43 pm

kwahoo wrote:
Sat Sep 21, 2019 7:05 pm
Steps to run the code:
1. Install pyopenvr, and sdl + opengl python packages
2. Run SteamVR
3. Paste the code into FreeCAD

Known issues:
1. The code renders two exactly same pictures - wrong buffer configuration?
2. Head tracking is not implemented yet
3. SoFrustumCamera needs further investigation. I'm trying to get necessary variables for the camera from IVRSystem::GetProjectionMatrix and IVRSystem::GetEyeToHeadTransform matrices, but maybe there is a better way?

Future work:
1. Fix issues
2. Remove SDL dependency, use QOpenGLWidget (?)
3. Port to C++
4. Add basic navigation
5. Add controllers support (for navigation)
6. Move to OpenXR
@kwahoo
Fantastic!
Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily. You can also create a Roadmap etc...
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
kwahoo
Posts: 235
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Postby kwahoo » Mon Sep 23, 2019 7:03 pm

Kunda1 wrote:
Sat Sep 21, 2019 7:43 pm
Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily.
Done. I updated the original post.

At this moment mostly(at least I hope so...) correct rendering with rotation and position tracking is possible.

I found the culprit of rendering the same image twice:
creating two scenes this way:

Code: Select all

rootScene = [SoSeparator()] * 2
...
for eye in range(2):
  m_sceneManager.setSceneGraph(rootScene[eye])
does not work. The second scene changes the first one.
schurik
Posts: 2
Joined: Wed Sep 11, 2019 10:35 pm

Re: OpenXR/OpenVR (virtual reality support)

Postby schurik » Mon Sep 23, 2019 10:09 pm

This sounds great.
Is it possible to run my Rift S on CATIA with this?
kwahoo
Posts: 235
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Postby kwahoo » Tue Sep 24, 2019 6:51 pm

schurik wrote:
Mon Sep 23, 2019 10:09 pm
Is it possible to run my Rift S on CATIA with this?
Rift S: Yes.
Catia: No, at least not with this script.

----
I have some issues to be ironed out, there is probably an error in cameras configuration (objects are too big, moving too fast). Update: this needs more testing, added a simple test file to the repository: a room with pillars and a table.

But meanwhile some performance teaser (click to see a bigger pic):
Image

To be honest, I was expecting worse fps;)
Last edited by kwahoo on Fri Sep 27, 2019 4:32 pm, edited 1 time in total.
User avatar
Kunda1
Posts: 5799
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Postby Kunda1 » Fri Sep 27, 2019 11:48 am

Sent another PR to soup up the README.
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
kwahoo
Posts: 235
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Postby kwahoo » Sun Sep 29, 2019 5:07 pm

Thanks!

There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.
User avatar
Kunda1
Posts: 5799
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Postby Kunda1 » Mon Sep 30, 2019 5:23 am

kwahoo wrote:
Sun Sep 29, 2019 5:07 pm
There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.
I would pose this as a question on the Open Issues subforum so it gets more exposure.
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
kwahoo
Posts: 235
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Postby kwahoo » Sat Nov 09, 2019 9:04 pm

I ported a basic OpenVR support to C++. It has the same functionality as the Python script.

The implementation can be found in this repository https://github.com/kwahoo2/FreeCAD Commit introducing OpenVR https://github.com/kwahoo2/FreeCAD/comm ... b8ab0c1e28

Notes about compilation:
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

There is no OS-specific code in the implementation - it should run both on Linux and Windows. Mac has some beta SteamVR support, but I do not have any experience with it .

Running a scene:
1. Start SteamVR
2. Open a model in the FreeCAD
3. Run View->FreeCAD-OpenVR