Decouple external build configuration (Travis CI/YCM) from local settings

[PeterBowman]

I've set up CI builds for asibot-hmi with this configuration; note that this project depends on kinematics-dynamics, hence an additional git clone step. To avoid building KDL, which is a dependency for kinematics-dynamics itself, I had to configure it appropriately:

cmake .. -DENABLE_TrajectoryLib:BOOL=OFF \
         -DENABLE_KdlVectorConverterLib:BOOL=OFF \
         -DENABLE_BasicCartesianControl:BOOL=OFF \
         -DENABLE_CartesianControlServer:BOOL=OFF \
         -DENABLE_CartesianControlClient:BOOL=OFF \
         -DENABLE_KdlSolver:BOOL=OFF \
         -DENABLE_BasicTwoLimbCartesianControl:BOOL=OFF \
         -DENABLE_TwoLimbCartesianControlServer:BOOL=OFF \
         -DENABLE_tests:BOOL=OFF

First remark: any options irrelevant to my project could've been set to OFF/FALSE in kinematics-dynamics' root CMakeLists.txt file. Therefore, I'd just concentrate on the funcionalities I'm really interested in, not the other way around.

Then, I configured periodic cron jobs targeted at discovering issues linked with the development of YARP. However, my latest build errored because of recent changes to kinematics-dynamics (CI logs), which happened to introduce a new dependency on KDL. This forced me to take a look upstream and update my .travis.yml accordingly: roboticslab-uc3m/asibot-hmi@9b1959d.

As a second remark, one must always keep an eye on each upstream repo because of new dependencies or changes in CMake configuration. This and the previous remark could be easily extended to YCM-like repos (e.g. teo-main) for the same reasons.

Proposal:

• set all CMake options to OFF (FALSE)
• recommend using GUI utilities (ccmake, cmake-gui) in install guides for ease of initial CMake configuration phase
• in downstream projects (WRT CI configuration files and YCM-specific build modules), list any required options and set them as required; in other words, don't expect foreign CMakeLists.txt files, which are easily subject to changes, to provide you the desired configuration

Regarding the last point and YCM projects, the CONFIGURE_COMMAND parameter to ycm_ep_helper should come in handy (docs).

[PeterBowman]

In order to alleviate configuration efforts (particularly with installation guides in mind), we could avoid passing several -Dxxx CL parameters (one per CMake option, e.g. -DENABLE_fancy_program and so on) by introducing a single -Cxxx parameter pointing at the most convenient .cmake intended for initializing the cache during the first run of cmake. Examples at robotology/yarp: CL invocation, initial-cache.cmake.

[PeterBowman]

Another ridiculous example (scroll right): roboticslab-uc3m/kinematics-dynamics@d5a9599.

[PeterBowman]

I did some CMake refactorization work in a bunch of repos. Namely:

• Refactor CMake files kinematics-dynamics#14
• Refactor CMake files yarp-devices#26
• Refactor CMake files openrave-yarp-plugins#56

Setting all options to OFF may create a twofold problem:

• If not handled correctly, CMake could be asked to process an export set with no actual targets linked to it. See CMake export system crashes when no export set available project-generator#19 and related issues.
• In order to prevent from building targets if their local dependencies are disabled, I resorted to using cmake_dependent_option() and the DEPENDS parameter of yarp_prepare_plugin() (examples: one, two). However, users might get tired of switching options to ON with ccmake and repeatedly configuring the project - relevant options won't show up until all their dependencies are previously enabled by hand.

[PeterBowman]

@jgvictores suggests to keep all/most options enabled and warn the user if it cannot get built for whatever reason (missing package, unsatisfied dependencies, etc.).

[jgvictores]

Yes, essentially all enabled, and automatically disabling (without breaking, just messages/warnings) if requirements are not met.

[PeterBowman]

Warnings may be annoying, but help to keep users aware of which components could not be successfully configured. Setting everything to OFF essentially hides the functionality of a repo.

Idea: try the FeatureSummary CMake module. It was adopted some time ago in amor-api, then in project-generator. Devs may use it to register all available CMake options and provide short descriptions. Then, CMake generates a report after the configuration step. Example output of amor-api:

-- YCM found in /usr/local/share/YCM.
-- AMOR API version: 0.2.1 (0.2.1)
-- Boost version: 1.58.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   chrono
--   system
--   atomic
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 
-- 
-- The following features have been enabled:

 * AMOR_API_extended , Advanced AMOR API functions.

-- The following REQUIRED packages have been found:

 * Eigen3
 * Boost

-- The following features have been disabled:

 * AMOR_API_cartesian_rate , cartesian_rate program (AMOR+YARP).

-- The following OPTIONAL packages have not been found:

 * Doxygen

-- Configuring done
-- Generating done
-- Build files have been written to: /home/bartek/wingit/amor-api/build-wsl

Starting from The following features have been enabled, we see the result of invoking feature_summary(). Moreover, it lists satisfied and missing package dependencies (find_package() calls).

[PeterBowman]

YARP devices already provide some sort of summarization mechanism, check this cmake .. sample run:

--  --- plugin AmorCartesianControl: disabled
--  +++ plugin AsibotSolver: enabled
--  +++ plugin BasicCartesianControl: enabled
--  --- plugin BasicTwoLimbCartesianControl: disabled (dependencies unsatisfied)
--  +++ plugin CartesianControlClient: enabled
--  +++ plugin CartesianControlServer: enabled
--  +++ plugin KdlSolver: enabled
--  +++ plugin TwoLimbCartesianControlServer: enabled

[PeterBowman]

keep all/most options enabled and warn the user if it cannot get built for whatever reason (missing package, unsatisfied dependencies, etc.)

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?) and then use the corresponding _FOUND variables in cmake_dependent_option()/yarp_prepare_plugin() (e.g. AmorCartesianControl/CMakeLists.txt@kinematics-dynamics).

[jgvictores]

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?)

Advantage: Faster and cleaner output of cmake.
Disadvantage: Goes a bit against philosophy of everything near where it is used...

Seeing the disadvantage... Maybe only in the case a dep is detected to be used on more than one component, then move up one level? I have no idea....

[PeterBowman]

Idea: place find_package() calls before the opening if() clause in cases such as:

yarp_prepare_plugin(AmorCartesianControl
                    CATEGORY device
                    TYPE roboticslab::AmorCartesianControl
                    INCLUDE AmorCartesianControl.hpp
                    DEFAULT ON
                    DEPENDS ENABLE_KinematicRepresentationLib)

if(NOT SKIP_AmorCartesianControl)

find_package(AMOR_API REQUIRED)

# ...

Obviously, the REQUIRED parameter should be removed. Then, custom code (more ifs, message(), etc.) would take care of warning the user about unsatisfied deps and reflecting that in the GUI.

[PeterBowman]

Working example at roboticslab-uc3m/kinematics-dynamics@14cf9cb. Template:

find_package(3rdparty_pkg QUIET)

if(NOT 3rdparty_pkg_FOUND AND (NOT DEFINED ENABLE_MyComponent OR ENABLE_MyComponent))
    message(WARNING "3rdparty_pkg package not found, disabling MyComponent")
endif()

# same with yarp_prepare_plugin()
cmake_dependent_option(ENABLE_MyComponent "Enable/disable MyComponent" ON
                       3rdparty_pkg_FOUND OFF)

if(ENABLE_MyComponent) # 'NOT SKIP_MyComponent' for YARP devices
    # ...
else()
    set(ENABLE_MyComponent OFF CACHE BOOL "Enable/disable MyComponent" FORCE)
endif()

Travis YAML files may be updated so that local repository options are now omitted (see .travis.yml).

Regarding non-local CMake options (as described in the first comment in this issue, i.e. remote repos pulled via .travis.yml with lots of disabled components), we'll want to provide a means to disable all/most non-vital options. The reason is that downstreams should not be forced to compile unnecesary programs and libraries, especially whenever we just want to get access to a header file. To achieve that, the initial-cache.cmake way may be a valid solution (#18 (comment)).

[PeterBowman]

Expanding on my previous comment, such refactorization is only intended for devices/programs/libraries that depend on REQUIRED 3rd-party packages - except YARP. As spoken with @jgvictores, YARP itself should be considered a hard dependency in our repos, that is, we'll always instruct CMake to search for it with find_package(YARP REQUIRED) (note the second parameter). I'd go a bit further and pull this call to the root CMakeLists.txt, before traversing any subdirectories.

[PeterBowman]

Proposed roadmap (I'll replicate and expand this in the issue description if nobody opposes):

• Apply improvements explained in Decouple external build configuration (Travis CI/YCM) from local settings #18 (comment) in local CMake list files, and simplify the option list passed to the cmake .. command in local .travis.yml. All options (except ENABLE_coverage and, perhaps, ENABLE_tests (see Look for system-installed googletest for use in Travis builds, drop embedded copies #41)) set to ON.
• Create an initial-cache.cmake in scripts/ (Decouple external build configuration (Travis CI/YCM) from local settings #18 (comment)) with all (relevant) options set to OFF, use this in foreign .travis.yml files. See last paragraph in previous comment for reasons.
• Use a single find_package(YARP REQUIRED) call in root CMakeLists.txt - except wherever a specific version of YARP is needed.

TODO: I need to check this ideas on a superbuild-like repo, especially regarding CMake options and BuildPkg.cmake files.

[PeterBowman]

Another TODO: if I place find_package(YARP REQUIRED) at the top, and then proceed as described earlier (3rd-party dep) with a YARP version that is not available on my system (e.g. find_package(YARP 2.3.70 QUIET)), what will happen with directories processed after that which depend on previous (supported) YARP releases?

(I was thinking about certain YARP devices in openrave-yarp-plugins)

[jgvictores]

I think I've found an issue with the current proposal.

Looking at these lines, cmake always goes through find_package(ARAVIS QUIET). Now, if we look at FindARAVIS.cmake, it has a find_package(GLib REQUIRED) which makes everything break, always, even if we are not using AravisGigE... :-(

[PeterBowman]

More examples:

• unsuitable Python version: roboticslab-uc3m/yarp-devices@b6fe2b9
• unsuitable YARP version: roboticslab-uc3m/openrave-yarp-plugins@18291fa

[PeterBowman]

This mechanism (#18 (comment)) is not aware of 3rd-party deps being available upon successive configure runs. Example:

• make sure that orocos_kdl is not installed on system nor registered at ~/.cmake/packages
• clone and configure https://github.com/roboticslab-uc3m/kinematics-dynamics/
• several components (e.g. KdlSolver) should be automatically disabled (and respond with a warning when manually switched to ON)
• configure and install orocos_kdl
• run cmake on kinematics-dynamics
  • expected: previously disabled options (depending on orocos_kdl) should be now enabled
  • result: nothing changes
• (we should not be here, but previous step failed...) enable KdlSolver via ccmake
  • expected: ...well, it's set to ON in the GUI
  • result: upon pressing c (configure), KdlSolver is switched back to OFF with no warning

[PeterBowman]

YARP itself should be considered a hard dependency in our repos, that is, we'll always instruct CMake to search for it with find_package(YARP REQUIRED) (note the second parameter)

Example: roboticslab-uc3m/yarp-devices@2fdc802.

[PeterBowman]

#18 (comment):

Idea: try the FeatureSummary CMake module.

I think it's already supported in custom YARP devices (ref, available since 2.3.68.1), that is, add a feature_summary(...) call at the end of root CMakeLists.txt and output should be automatically generated along with a list of available components.

[PeterBowman]

Another example (test two conditions): roboticslab-uc3m/vision@f5c89fb.

[PeterBowman]

Note: new YARP components may require additional attention when working with RL projects depending upon each other, see roboticslab-uc3m/kinematics-dynamics@273ac97 and #65.

[PeterBowman]

TODO: reflect this new dependency+warning mechanism in project-generator and repositories derived from this one (mainly tools and asibot-hmi).

[PeterBowman]

More ideas: robotology/yarp@758d23c (Plugins not enabled due to missing dependencies are now shown in ccmake andcmake-gui together with a list of dependencies that are not satisfied.).

[PeterBowman]

Somewhat a follow-up to #48 motivated by roboticslab-uc3m/openrave-yarp-plugins#91: if we just need a header file (usually, a YARP device interface) or a CMake module (rare) from a dependency, then clone/download, run cmake .., skip the compile part altogether and adjust the <project>_DIR variable.

[PeterBowman]

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?)

YARP itself should be considered a hard dependency in our repos

For future reference, this was done at #54.

[PeterBowman]

(#18 (comment))

This mechanism (...) is not aware of 3rd-party deps being available upon successive configure runs.

YARP v3.0.x suffers from this same problem, too, after robotology/yarp@758d23c was introduced. Looks like the final part of my proposed solution (#18 (comment)), which also resembles what YARP aims to achieve, is colliding with cmake_dependent_option's internals called either explicitly or from within yarp_prepare_plugin():

set(ENABLE_MyComponent OFF CACHE BOOL "Enable/disable MyComponent" FORCE)

[PeterBowman]

Note for travelers:

if(${var} MATCHES "^${var}$")

is a legacy way of saying:

if(NOT DEFINED ${var})

Found in CMakeDependentOption.cmake.

[PeterBowman]

CMake 3.13 introduces policy CMP0077. Not sure if we want this new behavior, though.

[PeterBowman]

This problem has been approached in some valid ways that are close enough to the intended result. I'm going to close it at last. A few remarks regarding what was discussed here:

• initial-cache.cmake: doable, but we don't sacrifice that much thanks to the adoption of ccache in CI builds
• feature_summary(): doable, but we tend to minimize console output, and YARP devices are already logged (this can be disabled, though)
• find_package(): I was considering the removal of the QUIET option; sadly, config-type packages (as opposed to find modules) are a bit too verbose on failure
• Decouple external build configuration (Travis CI/YCM) from local settings #18 (comment): not perfect, but it has served well all these years

Therefore, the original issue was already addressed quite a while ago. We ended up spreading boilerplate as detailed in previous comments (e.g. roboticslab-uc3m/vision@f5c89fb). Some people choose to see the ugliness in this code. The disarray. I choose to see the practicality. To believe there is an order to our CMake list files, a purpose.

[PeterBowman]

CMake 3.19 introduced presets which mostly behave like a JSON-formatted initial cache.