Skip to content

Commit

Permalink
Merge pull request #1456 from SamFlt/fix_sinatrack
Browse files Browse the repository at this point in the history 
Modifications for the upcoming Render-Based Tracking submodule
  • Loading branch information
@fspindle
fspindle authored Sep 6, 2024
2 parents 07c8be1 + 983d558 commit 3704e60
Show file tree
Hide file tree
Showing 28 changed files with 943 additions and 302 deletions.
213 changes: 143 additions & 70 deletions doc/tutorial/rendering/tutorial-panda3d.dox
View file
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.

- Now to build ViSP with Panda3D support when `.dmg` file `Panda3D-1.11.0-py3.9.dmg` is installed, you can just
run cmake as usual. Note that PCL is not compatible with Panda3D, that's why we disable here PCL usage
(see \ref tutorial-panda3d-issue-macOS).
(see \ref tutorial-panda3d-issue-segfault-macOS).
\code{.sh}
$ cd $VISP_WS/visp-build
$ cmake ../visp -DUSE_PCL=OFF
Expand All @@ -165,31 +165,81 @@ vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.

- Installer are available for Windows browsing the [download](https://www.panda3d.org/download/) page.

\section tutorial-panda3d-usage Rendere based on Panda3D usage
\section tutorial-panda3d-usage Using Panda3D for rendering

An example that shows how to exploit Panda3D in ViSP to render a color image with support for textures and lighting, a
depth image, normals in world space and in camera space is given in tutorial-panda3d-renderer.cpp.
depth image, normals in object space and in camera space is given in tutorial-panda3d-renderer.cpp.

Here you will find the code used to create the renderer:

To start rendering, we first instanciate a vpPanda3DRendererSet. This object allows to render multiple modalities (color, depth, etc.) in a single pass.
To add different rendering modalities, we will use subclasses that will be registered to the renderer set.
Internally, each sub renderer has its own scene: the renderer set synchronizes everything when the state changes (i.e, an object is added, an object is moved or the camera parameters change.)

A panda3D renderer should be instanciated with a vpPanda3DRenderParameters object. This object defines:
- The camera intrinsics (see vpCameraParameters): As of now, Only parameters for a distortion free model are supported
- The image resolution
- The near and far clipping plane values. Object parts that are too close (less than the near clipping value) or too far (greater than the far clipping value) will not be rendered.

The creation of the renderer set can be found below
\snippet tutorial-panda3d-renderer.cpp Renderer set

Here you will find the code used to create the sub renderers:
To actually render color, normals, etc., we need to define subrenderers:
\snippet tutorial-panda3d-renderer.cpp Subrenderers init

Here you will find the code used to add the sub renderers to the main renderer:
The different subrenderers are:

- vpPanda3DGeometryRenderer instances allow to retrieve 3D information about the object: these are the surface normals in the object or camera frame, as well as the depth information.
- vpPanda3DRGBRenderer objects perform the traditional color rendering. Lighting interaction can be disable, as is the case for the second renderer (diffuse only).
- Post processing renderers, such as vpPanda3DLuminanceFilter, vpPanda3DCanny, operate on the output image of another renderer. They can be used to further process the output data and can be chained together.
In this case, the chain vpPanda3DLuminanceFilter -> vpPanda3DGaussianBlur -> vpPanda3DCanny will perform a canny edge detection (without hysteresis) on a blurred, grayscale image.

For these subrenderers to actually be useful, they should be added to the main renderer:
\snippet tutorial-panda3d-renderer.cpp Adding subrenderers

\warning Once they have been added, a call to vpPanda3DBaseRenderer::initFramework() should be performed. Otherwise, no rendering will be performed and objects will not be loaded.

Here you will find the code used to configure the scene:
\snippet tutorial-panda3d-renderer.cpp Scene configuration

We start by loading the object to render with vpPanda3DBaseRenderer::loadObject, followed by vpPanda3DBaseRenderer::addNodeToScene.
For the Color-based renderer, we add lights to shade our object. Different light types are supported, reusing the available Panda3D features.

Once the scene is setup, we can start rendering. This will be performed in a loop.

The first step shown is the following:
\snippet tutorial-panda3d-renderer.cpp Updating render parameters

Each frame, we compute the values of the clipping planes, and update the rendering properties. This will ensure that the target object is visible.
Depending on your use case, this may not be necessary.

Once this is done, we can call upon Panda3D to render the object with
\snippet tutorial-panda3d-renderer.cpp Render frame

\note Note that under the hood, all subrenderers rely on the same Panda3D "framework": calling renderFrame on one will call it for the others.

To use the renders, we must convert them to ViSP images.
To do so, each subrenderer defines its own *getRender* method, that will perform the conversion from a panda texture to the relevant ViSP data type.

For each render type, we start by getting the correct renderer via the vpPanda3DRendererSet::getRenderer, then call its *getRender* method.
\snippet tutorial-panda3d-renderer.cpp Fetch render

Now that we have the retrieved the images, we can display them. To do so, we leverage utility functions defined beforehand (see the full code for more information). This may be required in cases where the data cannot be directly displayed.
For instance, normals are encoded as 3 32-bit floats, but displays require colors to be represented as 8-bit unsigned characters.
The same goes for the depth render, which is mapped back to the 0-255 range, although its value unbound.
\snippet tutorial-panda3d-renderer.cpp Display

Finally, we use the snippet below to move the object, updating the scene.
To have a constant velocity, we multiply the displacement by the time that has elapsed between the frame's start and end.
\snippet tutorial-panda3d-renderer.cpp Move object

\section tutorial-panda3d-full-code Tutorial full code

The full code of tutorial-panda3d-renderer.cpp is given below.
\include tutorial-panda3d-renderer.cpp

\section tutorial-panda3d-run Execute the tutorial
\section tutorial-panda3d-run Running the tutorial

- Once ViSP is build, you may run the tutorial by:
- Once ViSP is built, you may run the tutorial by:
\code{.sh}
$ cd $VISP_WS/visp-build
$ ./tutorial/ar/tutorial-panda3d-renderer
Expand All @@ -201,67 +251,90 @@ The full code of tutorial-panda3d-renderer.cpp is given below.
\endhtmlonly

\section tutorial-panda3d-issue Known issues
\subsection tutorial-panda3d-issue-macOS Known issue on macOS

- Segfault: `:framework(error): Unable to create window`
```
% ./tutorial-panda3d-renderer
Initializing Panda3D rendering framework
Known pipe types:
CocoaGLGraphicsPipe
(all display modules loaded.)
:framework(error): Unable to create window.
zsh: segmentation fault ./tutorial-panda3d-renderer
```
This issue is probably due to `EIGEN_MAX_ALIGN_BYTES` and `HAVE_PNG` macro redefinition that occurs when building ViSP with Panda3D support:
```
$ cd visp-build
$ make
...
[100%] Building CXX object tutorial/ar/CMakeFiles/tutorial-panda3d-renderer.dir/tutorial-panda3d-renderer.cpp.o
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:40:9: warning: 'HAVE_PNG' macro redefined [-Wmacro-redefined]
#define HAVE_PNG 1
\subsection tutorial-panda3d-issue-library-macOS Library not loaded: libpanda.1.11.dylib

This error occurs on macOS.

```
% cd $VISP_WS/visp-build/tutorial/ar/
% ./tutorial-panda3d-renderer
dyld[1795]: Library not loaded: @loader_path/../lib/libpanda.1.11.dylib
Referenced from: <0D61FFE0-73FA-3053-8D8D-8912BFF16E36> /Users/fspindle/soft/visp/visp_ws/test-pr/visp-SamFlt/visp-build/tutorial/ar/tutorial-panda3d-renderer
Reason: tried: '/Users/fspindle/soft/visp/visp_ws/test-pr/visp-SamFlt/visp-build/tutorial/ar/../lib/libpanda.1.11.dylib' (no such file)
zsh: abort ./tutorial-panda3d-renderer
```

It occurs when you didn't follow carefully the instructions mentionned in \ref tutorial-panda3d-install-macos section.

A quick fix is to add the path to the library in `DYLD_LIBRARY_PATH` env var:
```
$ export DYLD_LIBRARY_PATH=/Library/Developer/Panda3D/lib:$DYLD_LIBRARY_PATH
```

\subsection tutorial-panda3d-issue-segfault-macOS Segfault: :framework(error): Unable to create window

This error occurs on macOS.

```
% cd $VISP_WS/visp-build/tutorial/ar/
% ./tutorial-panda3d-renderer
Initializing Panda3D rendering framework
Known pipe types:
CocoaGLGraphicsPipe
(all display modules loaded.)
:framework(error): Unable to create window.
zsh: segmentation fault ./tutorial-panda3d-renderer
```
This issue is probably due to `EIGEN_MAX_ALIGN_BYTES` and `HAVE_PNG` macro redefinition that occurs when building ViSP with Panda3D support:
```
$ cd visp-build
$ make
...
[100%] Building CXX object tutorial/ar/CMakeFiles/tutorial-panda3d-renderer.dir/tutorial-panda3d-renderer.cpp.o
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:40:9: warning: 'HAVE_PNG' macro redefined [-Wmacro-redefined]
#define HAVE_PNG 1
^
/opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:53:9: note: previous definition is here
#define HAVE_PNG
^
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:64:9: warning: 'HAVE_ZLIB' macro redefined [-Wmacro-redefined]
#define HAVE_ZLIB 1
^
/opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:55:9: note: previous definition is here
#define HAVE_ZLIB
^
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:432:9: warning: 'EIGEN_MAX_ALIGN_BYTES' macro redefined [-Wmacro-redefined]
#define EIGEN_MAX_ALIGN_BYTES MEMORY_HOOK_ALIGNMENT
^
/opt/homebrew/include/eigen3/Eigen/src/Core/util/ConfigureVectorization.h:175:11: note: previous definition is here
#define EIGEN_MAX_ALIGN_BYTES EIGEN_IDEAL_MAX_ALIGN_BYTES
^
/opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:53:9: note: previous definition is here
#define HAVE_PNG
^
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:64:9: warning: 'HAVE_ZLIB' macro redefined [-Wmacro-redefined]
#define HAVE_ZLIB 1
^
/opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:55:9: note: previous definition is here
#define HAVE_ZLIB
^
In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
$VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:432:9: warning: 'EIGEN_MAX_ALIGN_BYTES' macro redefined [-Wmacro-redefined]
#define EIGEN_MAX_ALIGN_BYTES MEMORY_HOOK_ALIGNMENT
^
/opt/homebrew/include/eigen3/Eigen/src/Core/util/ConfigureVectorization.h:175:11: note: previous definition is here
#define EIGEN_MAX_ALIGN_BYTES EIGEN_IDEAL_MAX_ALIGN_BYTES
^
3 warnings generated.
[100%] Linking CXX executable tutorial-panda3d-renderer
[100%] Built target tutorial-panda3d-renderer
```
The work around consists in disabling `PCL` usage during ViSP configuration
```
$ cd $VISP_WS/visp-build
$ cmake ../visp -DUSE_PCL=OFF
$ make -j$(sysctl -n hw.logicalcpu)
```
3 warnings generated.
[100%] Linking CXX executable tutorial-panda3d-renderer
[100%] Built target tutorial-panda3d-renderer
```
The work around consists in disabling `PCL` usage during ViSP configuration
```
$ cd $VISP_WS/visp-build
$ cmake ../visp -DUSE_PCL=OFF
$ make -j$(sysctl -n hw.logicalcpu)
```

*/
68 changes: 29 additions & 39 deletions modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,17 +71,18 @@ class VISP_EXPORT vpPanda3DBaseRenderer
* Will also perform the renderer setup (scene, camera and render targets)
*/
virtual void initFramework();
virtual void initFromParent(std::shared_ptr<PandaFramework> framework, PointerTo<WindowFramework> window);
virtual void initFromParent(const vpPanda3DBaseRenderer &renderer);

/**
* @brief
*
* @param framework
* @param window
*/
void initFromParent(std::shared_ptr<PandaFramework> framework, std::shared_ptr<WindowFramework> window);


virtual void beforeFrameRendered() { }
virtual void renderFrame();
virtual void afterFrameRendered()
{
GraphicsOutput *mainBuffer = getMainOutputBuffer();
if (mainBuffer != nullptr) {
m_framework->get_graphics_engine()->extract_texture_data(mainBuffer->get_texture(), mainBuffer->get_gsg());
}
}

/**
* @brief Get the name of the renderer
Expand All @@ -90,6 +91,8 @@ class VISP_EXPORT vpPanda3DBaseRenderer
*/
const std::string &getName() const { return m_name; }

void setName(const std::string &name) { m_name = name; }

/**
* @brief Get the scene root
*
Expand All @@ -101,31 +104,7 @@ class VISP_EXPORT vpPanda3DBaseRenderer
*
* @param params the new rendering parameters
*/
virtual void setRenderParameters(const vpPanda3DRenderParameters &params)
{
unsigned int previousH = m_renderParameters.getImageHeight(), previousW = m_renderParameters.getImageWidth();
bool resize = previousH != params.getImageHeight() || previousW != params.getImageWidth();

m_renderParameters = params;

if (resize) {
for (GraphicsOutput *buffer: m_buffers) {
//buffer->get_type().is_derived_from()
GraphicsBuffer *buf = dynamic_cast<GraphicsBuffer *>(buffer);
if (buf == nullptr) {
throw vpException(vpException::fatalError, "Panda3D: could not cast to GraphicsBuffer when rendering.");
}
else {
buf->set_size(m_renderParameters.getImageWidth(), m_renderParameters.getImageHeight());
}
}
}

// If renderer is already initialized, modify camera properties
if (m_camera != nullptr) {
m_renderParameters.setupPandaCamera(m_camera);
}
}
virtual void setRenderParameters(const vpPanda3DRenderParameters &params);

/**
* @brief Returns true if this renderer process 3D data and its scene root can be interacted with.
Expand All @@ -143,6 +122,15 @@ class VISP_EXPORT vpPanda3DBaseRenderer
*/
int getRenderOrder() const { return m_renderOrder; }

void setRenderOrder(int order)
{
int previousOrder = m_renderOrder;
m_renderOrder = order;
for (GraphicsOutput *buffer: m_buffers) {
buffer->set_sort(buffer->get_sort() + (order - previousOrder));
}
}

/**
* @brief Set the camera's pose.
* The pose is specified using the ViSP convention (Y-down right handed).
Expand Down Expand Up @@ -206,8 +194,10 @@ class VISP_EXPORT vpPanda3DBaseRenderer
* @param name name of the node that should be used to compute near and far values.
* @param near resulting near clipping plane distance
* @param far resulting far clipping plane distance
* @param fast Whether to use the axis align bounding box to compute the clipping planes.
* This is faster than reprojecting the full geometry in the camera frame
*/
void computeNearAndFarPlanesFromNode(const std::string &name, float &near, float &far);
void computeNearAndFarPlanesFromNode(const std::string &name, float &near, float &far, bool fast);

/**
* @brief Load a 3D object. To load an .obj file, Panda3D must be compiled with assimp support.
Expand Down Expand Up @@ -250,6 +240,8 @@ class VISP_EXPORT vpPanda3DBaseRenderer

virtual GraphicsOutput *getMainOutputBuffer() { return nullptr; }

virtual void enableSharedDepthBuffer(vpPanda3DBaseRenderer &sourceBuffer);

protected:

/**
Expand All @@ -272,16 +264,14 @@ class VISP_EXPORT vpPanda3DBaseRenderer
*/
virtual void setupRenderTarget() { }


const static vpHomogeneousMatrix VISP_T_PANDA; //! Homogeneous transformation matrix to convert from the Panda coordinate system (right-handed Z-up) to the ViSP coordinate system (right-handed Y-Down)
const static vpHomogeneousMatrix PANDA_T_VISP; //! Inverse of VISP_T_PANDA


protected:
const std::string m_name; //! name of the renderer
std::string m_name; //! name of the renderer
int m_renderOrder; //! Rendering priority for this renderer and its buffers. A lower value will be rendered first. Should be used when calling make_output in setupRenderTarget()
std::shared_ptr<PandaFramework> m_framework; //! Pointer to the active panda framework
std::shared_ptr<WindowFramework> m_window; //! Pointer to owning window, which can create buffers etc. It is not necessarily visible.
PointerTo<WindowFramework> m_window; //! Pointer to owning window, which can create buffers etc. It is not necessarily visible.
vpPanda3DRenderParameters m_renderParameters; //! Rendering parameters
NodePath m_renderRoot; //! Node containing all the objects and the camera for this renderer
PointerTo<Camera> m_camera;
Expand Down
Loading

0 comments on commit 3704e60

Please sign in to comment.