-
Notifications
You must be signed in to change notification settings - Fork 256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable SDK for ROS2 on Scarthgap #1241
Conversation
@jiaxshi @robwoolley can you review this? It has been tested on a Raspberry Pi5 and is functional. |
Thanks, I was at ROSCon this week. I will take a look at it when I return home. |
set(BOOST_ROOT "${PCL_ROOT}/3rdParty/Boost") | ||
elseif(NOT BOOST_INCLUDEDIR) | ||
- set(BOOST_INCLUDEDIR "@Boost_INCLUDE_DIR@") | ||
+ set(BOOST_INCLUDEDIR "$ENV{OECORE_TARGET_SYSROOT}/usr/include") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @sgstreet
OECORE_TARGET_SYSROOT is defined and used in SDK environment. It should be null in yocto build time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I'm aware of this will be empty at yocto build time. My hope, untested at this point, is that non-sdk builds will set BOOST_INCLUDEDIR, EIGEN_ROOT and FLANN_ROOT correctly. Did you test this PR again one of your builds?
Can you suggest an different approach to prevent the leakage of the build directory into the SDK? I can remove this commit and update the sdk colcon build instructions if you want?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out that the building images works even when OECORE_TARGET_SYSROOT is empty but present in the PCLConfig.cmake file. But in the interest of making the least change 9f47588 removes the reference to OECORE_TARGET_SYSROOT. @robwoolley Thanks for the insights!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can use ${CMAKE_SYSROOT} here, it's a CMAKE variable and set in both yocto build env and SDK env.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
9f47588 removes the references to OECORE_TARGET_SYSROOT and works both when building images/sdk and when using the sdk. Is this sufficient?
ee3109e
to
2c28501
Compare
682617c
to
b99fe2b
Compare
I rebased onto the current scarthgap-next |
Hi all, great work on getting ROS2 to be part of the SDK! It looks like it's almost working on my build, but I do get these two build errors: rosidl-typesupport-interface.log.do_configure.txt For
And for
I wonder if this is because I'm using a non-standard ros_prefix ( |
@jdiez17 Could you provide the values of PYTHONPATH, AMENT_PREFIX_PATH and the colcon build command you used? All three of these interact to find the ROS2 build system components. |
@@ -25,11 +25,13 @@ export AMENT_PREFIX_PATH="${STAGING_DIR_HOST}${prefix};${STAGING_DIR_NATIVE}${pr | |||
|
|||
inherit cmake python3native | |||
|
|||
PYTHONPATH:prepend:class-nativesdk = "${STAGING_DIR_NATIVE}/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_BASEVERSION}/site-packages:" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PYTHONPATH:prepend:class-nativesdk = "${STAGING_DIR_NATIVE}/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_BASEVERSION}/site-packages:" | |
PYTHONPATH:prepend:class-nativesdk = "${STAGING_DIR_NATIVE}${ros_prefix}/lib/python${PYTHON_BASEVERSION}/site-packages:" |
I suspect this hardcoded path caused my build error. Currently running a build with this change, seems to get further. Will report back if it builds and I can compile a workspace with the SDK.
I didn't get to building a workspace with this yet. That error was from running |
@@ -38,3 +40,8 @@ EXTRA_OECMAKE:prepend:class-native = "\ | |||
-DCMAKE_PREFIX_PATH='${ros_prefix}' \ | |||
-DCMAKE_INSTALL_PREFIX:PATH='${ros_prefix}' \ | |||
" | |||
|
|||
EXTRA_OECMAKE:prepend:class-nativesdk = "\ | |||
-DCMAKE_PREFIX_PATH='${STAGING_DIR_NATIVE}/opt/ros/${ROS_DISTRO};${STAGING_DIR_NATIVE}${ros_prefix};${STAGING_DIR_NATIVE}${prefix}' \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't ${STAGING_DIR_NATIVE}/opt/ros/${ROS_DISTRO}
and ${STAGING_DIR_NATIVE}${ros_prefix}
the same? (except, of course, in situations where ros_prefix
is customized - like in my build).
-DCMAKE_PREFIX_PATH='${STAGING_DIR_NATIVE}/opt/ros/${ROS_DISTRO};${STAGING_DIR_NATIVE}${ros_prefix};${STAGING_DIR_NATIVE}${prefix}' \ | |
-DCMAKE_PREFIX_PATH='${STAGING_DIR_NATIVE}${ros_prefix};${STAGING_DIR_NATIVE}${prefix}' \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure seem like a problem. I will try on my desktop
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does b99fe2b fix it?
Unfortunately, using This may be because |
Indeed ${ros_prefix} seems incorrect for
For reference:
|
@jdiez17 Seems a like a bigger problem with the ros_prefix preventing alternative install locations. Maybe we should create a new issue to explore how to fix this? |
@sgstreet Agreed. As discussed, this should not block merging this PR. The only thing I can say is that the hardcoded paths (as they are in the PR currently) should stay there for now. |
@jdiez17 Can you try something like this?
And this set ros_install_prefix in your build environment. I'm pretty sure the SDK ROS2 build tools are assuming some dependencies are present in their final location |
@sgstreet I can confirm that it works with your proposed patch (committed in my fork here). This builds the SDK successfully, but seems numpy is not included:
Not sure why numpy is needed for message generation, but will continue investigating. |
My bad, I will update the usage instructions, try adding:
To your colcon build command -cmake-args. I was unable to track down this problem, it seem like it is an issue with the numpy cmake module, but I could no track it down. You were trying to use the SDK to build a workspace correct? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{common} Add native sdk class extentions
This looks like the correct, simple solution for now. In the longer term, we may want to replace this with a variable in ros-distro.inc that automatically extends the recipes to include nativesdk if the recipe name is in the list.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{rolling} Patch fastrtps to use find TinyXML2
I'm fine with the changes, but it is unclear what problem this solves.
Can you provide more detail on why this was necessary to enable the SDK?
Newer versions of Bitbake also require the Upstream-Status field to be included in the header of the patch. Is this a change that would be applicable upstream or is it oe-specific? (https://docs.yoctoproject.org/contributor-guide/recipe-style-guide.html#patch-upstream-status)
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{common}{ros2} Ensure EXTRA_OECMAKE overrides correctly handle multiple cmake -D options
The change looks good. Thanks for tracking this down!
I don't think I understand the comment about making sure it handles multiple cmake -D options, could you elaborate on what the problem was and how this fixes that?
I also need to double-check whether STAGING_DIR_HOST is used and not STAGING_DIR_TARGET.
Do we need to set CMAKE_MODULE_PATH for native and nativesdk? Without it I worry we may be linking against Python modules provided by the host instead of the -native Python modules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
meta-ros-common/recipes-core/images/ros2-image-sdktest.bb:
I think having this alongside ros-image-sdktest.bb makes sense. We may want to rename the other image recipe ros1-image-sdktest.bb to avoid confusion for ROS 2 users.
I see that it includes some heavy hitters like OpenCV, Python, NumPy, Boost, etc. It looks like a good regression test for making sure that the SDK is working. Can we also document the steps needed to exercise the SDK?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
meta-ros-common/recipes-devtools/colcon/python3-colcon-core/0003-Force-shebang-to-usr-bin-env-python3.patch:
Nice find, looks good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{humble} Use find package for yaml to prevent sdk build failures
I wonder if we should be using yaml_vendor instead?
Signed-off-by: Stephen Street <[email protected]>
…rge installs Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
…configuration Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
4f60fc1
to
f60c234
Compare
I rebased onto the tip of scarthgap-next. |
ee5d4d3
to
db36783
Compare
echo "export PYTHON_SOABI=${PYTHON_SOABI}.so" >> ${S}/ros-sdk-env.sh | ||
echo "export CMAKE_PYTHON_SOABI=${PYTHON_SOABI}" >> ${S}/ros-sdk-env.sh | ||
echo "export PYTHON3_NUMPY_INCLUDE_DIR="'$OECORE_TARGET_SYSROOT'"/usr/lib/python${PYTHON_BASEVERSION}/site-packages/numpy/core/include" >> ${S}/ros-sdk-env.sh | ||
echo "export PYTHONWARNINGS=ignore" >> ${S}/ros-sdk-env.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we shouldn't silence Python warnings at the SDK level. Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. This is a hack to quite the numerous setuptools tools warnings. Not being a ptyhton expert I could not figure out the magic incantation to silence just the setuptool warning.
echo "export ROS_DISTRO=${ROS_DISTRO}" >> ${S}/ros-sdk-env.sh | ||
echo "export PYTHONPATH="'$OECORE_NATIVE_SYSROOT'"/usr/lib/python${PYTHON_BASEVERSION}/site-packages:"'$OECORE_TARGET_SYSROOT'"${ros_base_prefix}/lib/python${PYTHON_BASEVERSION}/site-packages" >> ${S}/ros-sdk-env.sh | ||
echo "export AMENT_PREFIX_PATH="'$OECORE_TARGET_SYSROOT'"${ros_base_prefix}" >> ${S}/ros-sdk-env.sh | ||
echo "export PYTHON_SOABI=${PYTHON_SOABI}.so" >> ${S}/ros-sdk-env.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PYTHON_SOABI
should not have a .so prefix:
builder@7f008748f080:/build$ bitbake-getvar PYTHON_SOABI -r rmw-implementation
WARNING: You have included the meta-virtualization layer, but 'virtualization' has not been enabled in your DISTRO_FEATURES. Some bbappend files may not take effect. See the meta-virtualization README for details on enabling virtualization support.
#
# $PYTHON_SOABI
# set /build/../work/src/ros/meta-ros2/classes/ros_ament_cmake.bbclass:15
# "cpython-${@d.getVar('PYTHON_BASEVERSION').replace('.', '')}${PYTHON_ABI}-${PYTHON_SOABI_ARCH}${PYTHON_SOABI_ARCH_SUFFIX}"
PYTHON_SOABI="cpython-312-aarch64-linux-gnu"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See CMAKE_PYTHON_SOABI at line 40. This value should be used in the --cmake-args.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand, but you're setting it to ${PYTHON_SOABI}
.so. The .so
needs to go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reread your message and now I see what you mean. Sorry about that! Just for my curiosity, what is the "plain" PYTHON_SOABI
used for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I picked this you from net somewhere as required. I do not understand of this environment variable effects python interpreter execution. @robwoolley has a difference approach which I also do not understand. I think we need clarity, idea?
inherit ros_distro_${ROS_DISTRO} | ||
inherit ${ROS_DISTRO_TYPE}_image | ||
|
||
IMAGE_FEATURES:append = " \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMAGE_FEATURES:append = " \ |
Remove no-op lines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep
ament-cmake-test \ | ||
ament-cmake-vendor-package \ | ||
ament-cmake-version \ | ||
foonathan-memory-staticdev \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This list is incomplete. At the very minimum we need python-cmake-module
, possibly others as well. At the moment I'm using the "kitchen sink" task list from previous versions of this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I notice that python-cmake-module is present in the target sysroot, which is why this might work.
One challenge that we have is in defining which packages are "host" dependencies and which are "target" dependencies. I need to do an audit of this, because I'm pretty sure that I have made this mistake when adding ROS_BUILD_DEPENDS and ROS_EXEC_DEPENDS in the various bbappends files to satisfy the builds.
The tricky party is that if you intend to do target-side builds then ament-cmake-* should be in the target list. However, for an SDK it probably should be only present in the host. I think if you are building an SDK for an image that supports target-side builds, then it should probably exist in both and the PYTHONPATH should be ordered to grab the host module first. (AFAIK)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to trim this down the the bare minimum. I think the correct approach is packagegroups for the SDK functionality. Which should be up next.
Signed-off-by: Stephen Street <[email protected]>
Signed-off-by: Stephen Street <[email protected]>
db36783
to
9c32b48
Compare
Here is the latest preview of the series: https://github.com/ros/meta-ros/commits/master-next-sdkfixes-latest/ |
Rebased and the commits into scarthgap-next. Would it be possible to open a new SDK for the envvar commit? 5efc734 |
Working on it just running some tests |
This PR enables OpenEmbedded SDK building and cross-compilation for Rolling, Jazzy and Humble. This PR is based on the work done in #988, #1203 and #1215. See these for more discussion
These changes allow demos, examples and examples_interfaces packages to be built using the resulting SDK. To build the SDK use:
$ bitbake ros2-image-sdktest -c do_populate_sdk
Install the resulting SDK, setup a ROS2 workspace containing demos, examples and example_interfaces and build the workspace using the follow script:
export ROS_VERSION=2
export ROS_PYTHON_VERSION=3
export ROS_DISTRO=${ROS_DISTRO:-rolling}
export PYTHONPATH=$OECORE_NATIVE_SYSROOT/usr/lib/python3.12/site-packages:$OECORE_TARGET_SYSROOT/opt/ros/rolling/lib/python3.12/site-packages
export AMENT_PREFIX_PATH=$OECORE_TARGET_SYSROOT/opt/ros/rolling
usr/share:$OECORE_TARGET_SYSROOT/opt/ros/rolling/share:$OECORE_TARGET_SYSROOT/opt/ros/rolling/lib/cmake"
export PYTHON_SOABI=cpython-312-aarch64-linux-gnu.so
colcon build \
--merge-install \
--install-base $PWD/install/arm64 \
--cmake-args \
" --no-warn-unused-cli" \
" -DCMAKE_TOOLCHAIN_FILE=$CMAKE_TOOLCHAIN_FILE" \
" -DCMAKE_STAGING_PREFIX=$PWD/install/arm64" \
" -DPYTHON_SOABI=$(basename -s so $PYTHON_SOABI)" \
" -DPython3_NumPy_INCLUDE_DIR=$OECORE_TARGET_SYSROOT/usr/lib/python3.12/site-packages/numpy/core/include" \
" -DBUILD_TESTING=OFF"
Install the $PWD/install/arm64 on the target device. Setup the device workspace:
$ . /opt/ros/rolling/setup.bash
$ . ./arm64/local_setup.bash
Run any of the examples to demonstrate the valid cross-compilation.
Co-developed-by: Romain Gimenez [email protected]
Co-developed-by: Jiaxing Shi [email protected]