Unified way of describing dependencies of a project

As a note to future discussion participants, there is high potential for this issue to suffer the same fate as #22619 (closed) and discussion threads to venture significantly beyond the main focus. This issue is already pretty huge in scope. If you would like to comment on aspects that are outside this focus, please do consider using or opening a separate issue and referring back to this one so that we can try to keep things actionable.

That aside, don't let that dissuade you from contributing your relevant thoughts. This topic is hard and I expect some of the things mentioned in the description will morph dynamically as discussions evolve.

The current state of affairs has effectively fused together several related, but (ought-to-be) independent concepts:

The dependency interface: the CMake targets, variables, and helper functions consumers use to link to the dependency. Also includes input variables to configure the interface. Examples:
1. Halide provides Halide::Halide for JIT programs as well as add_halide_library to simplify calling ahead-of-time generators.
2. LLVM provides no fixed targets and instead provides a mapping function llvm_map_components_to_libnames and variables LLVM_DEFINITIONS and LLVM_INCLUDE_DIRS.
3. Qt provides targets like Qt6::Widgets and honors input variables like QT_NO_CREATE_VERSIONLESS_FUNCTIONS.
The dependency origin: where to go to get the things a provider (see below) needs to provide the interface. Examples:
1. A local directory containing sources. (e.g. a git submodule)
2. A local directory containing binaries and CMake config scripts. (e.g. /usr or another CMAKE_PREFIX_PATH entry)
3. A remote repository (git, https, svn, etc.) containing any of the above.
The dependency provider: the actor responsible for resolving an origin (or a list of origins) to a loaded interface.
1. find_package in CONFIG mode
2. find_package in MODULE mode
3. FetchContent (or its more famous special case, add_subdirectory)
4. pkg-config, vcpkg, Conan, or another package manager
5. ad-hoc or generated code creating imported targets

Some notes on this taxonomy:

Some providers layer upon others.
- For instance, vcpkg stages everything in a common install prefix and then relies on find_package to import the packages.
Not all providers support all origin types.
Not all dependencies have consistent interfaces
- Not all dependencies have CMake builds
- Not all CMake dependencies have consistent build and install interfaces (#22687 is a step in the right direction)
Some providers behave badly and alter the interfaces, even when an upstream has established an interface.
- Conan and pkg-config are, in my experience, quite bad about this.
- Vcpkg does a better job, but can still introduce incompatibilities when the upstream package is broken and they have to patch around it.
- CMake's own providers are internally inconsistent in some places.
  - Loading the BLIS linear algebra library through FindBLAS is a completely different thing from loading it through FindPkgConfig, even from the same origin.

I hope this provides some utility towards structuring our thoughts on this.

A few more (looser) notes:

PkgConfig is fundamentally incapable of providing an interface that includes some CMake functions or whose targets contain generator expressions.
- This could be fixed upstream by adding something like our "vendor" objects, but I'm not sure whether the devs would like that.
- Library authors intending to support both PkgConfig and find_package(CONFIG) providers might need to split their library into two packages: LibCore (nothing but genex-free targets) and Lib (which depends on LibCore and extends its interface).
- FindPkgConfig will need to evolve to honor the interfaces of non-CMake dependencies that have first-party Find modules (at least)
I said nothing above on how to configure a provider, but there is a clear need for this, too.
- FetchContent is hard to override (but also supports the most origin types!)
I did not define a dependency spec or interface compatibility.
- A dependency spec is (at least) a name and a version that maps to an interface (as defined above)
- Interface compatibility is defined via Liskov Substitution, but ensuring it is the responsibility of each provider, not of CMake.
  - Note that this specifically allows a provider to supply a newer version, even if the name has changed, as long as it knows it will provide a wholly compatible interface (including target non-definition).
- Two interfaces can be mutually incompatible if loading them both at the same time would result in an error (again, the provider chain's responsibility).
I did not specify a (transitive) dependency resolution strategy:
- CMake probably needs to record an environment of loaded dependencies, regardless of provider. This will need to be directory-scoped.
- It should be up to each provider whether it can fulfill a given dependency request, given the current environment.
I did not specify an interface for querying a particular provider or chain of providers.
It might be useful to distinguish a dependency origin from a dependency repository
- If we do, then /usr would be a repository and /usr/lib/llvm-12/lib/cmake/llvm/LLVMConfig.cmake would be the origin (or any suitably specific parent of that path).
- This might be overkill.
Whatever the new mechanism is for loading dependencies, it should not take any inputs from the CMakeLists.txt besides the dependency spec and any arguments (i.e. input variables) to the interface. All that should be configured externally. Projects can provide defaults in some form that is impossible to hard-code.

The dependency interface: the CMake targets, variables, and helper functions consumers use to link to the dependency. Also includes input variables to configure the interface. Examples:

Halide provides Halide::Halide for JIT programs as well as add_halide_library to simplify calling ahead-of-time generators.

LLVM provides no fixed targets and instead provides a mapping function llvm_map_components_to_libnames and variables LLVM_DEFINITIONS and LLVM_INCLUDE_DIRS.

Qt provides targets like Qt6::Widgets and honors input variables like QT_NO_CREATE_VERSIONLESS_FUNCTIONS.

I would categorize these as dynamic dependencies which would need to be configured within CMakeLists.txt file (similar to static dependencies in pyproject.toml and dynamic ones in setup.py). I.e. the meta file should have the bare minimum, only allowing to pass the name, version and components, while any other fine-tuned configurations should be done in the CMakeLists file.

Not all dependencies have consistent interfaces

Hasn't cmake converged to using MODULE mode Find<Pkg> for the non-cmake cases.

Not all CMake dependencies have consistent build and install interfaces

Would this mean that it would work differently than the FetchContent/add_subdirectory and actually make an install in order to be included. This should be made explicit because this will affect both the consumer and the packager on how they design their cmake projects.

The main goal of this proposal is to unify find_package(), FetchContent and FindPkgConfig such that there is a single way of describing the dependencies of a project. This immediately gives the following consequences:

Simple solution: Put FetchContent and FindPkgConfig into a Find<Module>.cmake for find_package() to find. You can simply write a helper for that which takes e.g. a json file as an input? And really I don't see any need here to provider further functionality within CMake. Simply teach people to use find_package() correctly and have everything use targets defined within.

What could help here is a way to automatically define/create Find<Module>.cmake for a project depending on some common (json) format. Furthermore some way to add the find_package calls to a -Config.cmake would be nice

JSON seems the obvious leading candidate.

If JSON is used, the schema should allow a comment field in most objects. The official JSON format doesn't support comments, but being able to provide them would be desirable for human readability use cases. The comment field would never affect processing, it would only be for human information and recording arbitrary text.

Since this has to be a human readable format that supports comments, why not use TOML?

JSON is already used by presets, so using JSON for this as well has the advantage of consistency, but maybe it is something that is worth considering.

i have no experience with TOML, it looks like it doesn't support nesting attributes? Nesting would be convenient for components, for example. I consider JSON a poor choice for CMake presets, and also in this case. I much prefer YAML.

YAML does not require a science or engineering background, it is human readable and should be understandable to anyone familiar with plain text lists
It supports comments
its usage is widespread with libraries available for lots of programming languages
it is very common in the tooling business, users are familiar with YAML
IDE commonly provide native support for syntax highlighting and autocompletion, often additionally provide linter
it supports multiline values: strings, comments, lists, etc
character escaping is probably similar intrusive like JSON

i have no experience with TOML, it looks like it doesn't support nesting attributes?

It supports nesting, but it may be a little awkward to write/read:

[one]
something = 'random'

[one.two]
three = 4

is equivalent to

{
    "one":
    {
        "something": "random",
        "two":
        {
            "three": 4
        }
    }
}

I consider JSON a poor choice for CMake presets, and also in this case. I much prefer YAML.

YAML also works. As a CMake user I just think that using a format that has support for comments is better than working around the lack of comments support in JSON.

For example, when adopting the presets feature I constantly felt the need to comment or uncomment fields in the presets JSON just to test things out, but I couldn't do that, so I had to cut them out, which led to further problems because JSON also doesn't like trailing commas. Not a huge deal, but there are more user friendly formats out there.

On the other hand, it may be a little early to discuss the format used for this and I don't want to distract from more important aspects, I just got excited when I saw the issue.

Thanks for the thoughts. Yes, let's defer talking about the format until the rest of the topic has been explored. I think the file format won't largely influence that discussion, so we can leave it to the end.

I would say that the file format should NOT influence this discussion in any way. At the end, if a format will not fulfill the needs (comments, nesting, ...), it should simply be discarded. Of those remaining, pick one for coherency and/or wide usage in similar tech.

unassigned @craig.scott

mentioned in issue #24885 (closed)

JSON seems the obvious leading candidate.

If JSON is used, the schema should allow a comment field in most objects. The official JSON format doesn't support comments, but being able to provide them would be desirable for human readability use cases. The comment field would never affect processing, it would only be for human information and recording arbitrary text.

I think this should be changed in the main issue text. The case against comments in JSON is that that would not make it readable by arbitrary json parsers, which is the main point of having it machine readable. It should be replaced with a discussion of using either yaml or toml, or both.

One thing I don't see discussed is the frontend interface in the CMakeLists.txt file. I would say it could be like

project(META_FILE [/path/to/meta_file])
...
# option definition and stuff
...
MetaData_MakeAvailable()

The naming is horrible, but the idea is to have a workflow similar to FetchContent so that people can optimize configuration options, e.g. setting (cache) variables for the dependencies. This would future-proof in case we want this file to have more general metadata, e.g. dynamic versioning, which could be configured dynamically outside the the static meta file.

mentioned in merge request !9180 (closed)

mentioned in issue #25628

changed the description

mentioned in issue #25937

mentioned in issue #26771 (closed)

Unified way of describing dependencies of a project

Child items 0

Activity