Calling configure will now implicitly run init-repository when
appropriate. See further down below for what "appropriate" means.
All supported init-repository options can be passed to configure as
except for -mirror, -oslo, -berlin.
This includes useful options like -submodules, -no-resolve-deps and
-no-optional-deps.
When running configure on a qt5.git clone without any submodules
cloned, configure will exit with a helpful error message suggesting to
pass -init-submodules, so it automatically clones missing repositories.
This means cloning is opt-in, so that internet access is not done
implicitly.
The error message also suggests passing the -submodules option.
This will affect which submodules will be cloned by init-repository
and which submodules will be configured by configure.
In this case -submodules is effectively an alias of
init-repository's -module-subset for cloning purposes.
When calling configure a second time, without -init-submodules, on an
already configured repo, init-repository behavior is entirely skipped.
-submodules now accepts init-repository-style special values like
"essential", "addon", "all", "existing", "-deprecated" for the purpose
of cloning submodules. The values are then translated into actual repos
that should also be configured or skipped by configure.
The default subset of cloned submodules is currently the same one as
init-repository, "default", which clones 44 actively maintained
repositories as well as deprecated submodules.
If configure is called a second time WITH -init-submodules, it's the
same as calling init-repository --force to re-initialize submodules.
In this case passing something like
--submodules existing,<additional-submodules>
might make sense to add or remove submodules.
As a drive-by this also fixes the bug where you couldn't pass a
configure -- -DFOO=0
parameter to configure, because it got treated as '0>', redirecting
from a different stream than stdout, leading to empty content in the
file.
[ChangeLog][General][Build System] The configure script now implicitly
calls init-repository when appropriate and accepts init-repository
command line options.
Fixes: QTBUG-120030
Task-number: QTBUG-122622
Change-Id: Iedbfcbf0a87c8ee89e40d00b6377b68296a65a62
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
init-repository is now implemented using CMake + .sh / .bat scripts.
The intent behind the change is not to require Perl to checkout and
build Qt, because it can be troublesome to acquire on Windows and it
can also lead to issues during builds due to CMake picking up a Perl
distribution-shipped compiler.
All previous options were ported over like
- module-subset
- alternates
- etc.
A few new options were added:
- --resolve-deps / --no-resolve-deps
- --optional-deps / --no-optional-deps
- --verbose
and some other internal ones for testing reasons.
The new script does automatic resolving of dependencies
based on the depends / recommends keys in .gitmodules unless
--no-resolve-deps is passed.
So if you configure with --module-subset=qtsvg, the script will also
initialize qtbase.
If --no-optional-deps is passed, only required dependencies ('depends'
ky) will be included and optional dependencies ('recommends' key) will
be excluded.
The new script now has a new default behavior when calling
init-repository a second time with --force, without specifying a
--module-subset option. Instead of initializing all submodules, it
will just update the existing / previously initialized submodules.
It also understands a new module-subset keyword "existing", which
expands to the previously initialized submodules, so someone can
initialize an additional submodule by calling
init-repository -f --module-subset=existing,qtsvg
Implementation notes:
The overall code flow is init-repository -> cmake/QtIRScript.cmake
-> qt_ir_run_main_script -> qt_ir_run_after_args_parsed ->
qt_ir_handle_init_submodules (recursive) -> qt_ir_clone_one_submodule
with some bells and whistles on the side.
The command line parsing is an adapted copy of the functions
in qtbase/cmake/QtProcessConfigureArgs.cmake. We can't use those exact
functions because qtbase is not available when init-repository is
initially called, and force cloning qtbase was deemed undesirable.
We also have a new mechanism to detect whether init-repository was
previously called. The perl script used the existence of the qtbase
submodule as the check. In the cmake script, we instead set a custom
marker into the local repo config file.
Otherwise the code logic should be a faithful reimplementation of
init-repository.pl aside from some small things like logging and
progress reporting.
The pre-existing git cloning logic in QtTopLevelHelpers was not used
because it would not be compatible with the alternates option and I
didn't want to accidentally break the pre-existing code. Plus
init-repository is a bit opinionated about how it clones and checks
out repos.
The dependency collection and sorting logic uses the pre-existing code
though.
See follow up commit about implicitly calling init-repository when
qt5/configure is called and the repo was not initialized before.
[ChangeLog][General] init-repository was rewritten using CMake. Perl
is no longer required to initialize the qt5.git super repo.
Task-number: QTBUG-120030
Task-number: QTBUG-122622
Change-Id: Ibc38ab79d3fdedd62111ebbec496eabd64c20d2b
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The script produces the list of Qt submodules that are required to
build the submodules listed in the QT_BUILD_SUBMODULES variable.
Also the script supports the "collect all but skip repos" mode.
The repos that needs to be skipped should be specified by setting
BUILD_<repo>=FALSE.
The resulting list preserves the required build order.
Change-Id: Ia1cfe0ce013f80dc2793a0ba863a18789657e5a1
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Amir Masoud Abdol <amir.abdol@qt.io>
Developers that use a top-level qt5.git can use git submodule foreach
to run commands for each submodule. That is not available to
developers that maintain their own non-submodule based top-level
checkout using git worktrees.
Extend the QtSynchronizeRepo.cmake script that allows to sync
submodules / worktrees to also allow running foreach commands.
The CMake invocation to run a foreach would look like
cmake -DQT_FOREACH=TRUE "-DARGS=git pull -r" -P
cmake/QtSynchronizeRepo.cmake
But it's nicer to create a git-qt-foreach script in
/usr/local/bin/git-qt-foreach with the following contents
cmake -DQT_FOREACH=TRUE "-DARGS=$*" -P cmake/QtSynchronizeRepo.cmake
Make sure to chmod a+x it and invoke like
git qt-foreach git pull -r
Change-Id: I0ccd67facbfe702c6a872165cdfd9f9e6eae9043
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
In tqtc repos, the dependencies.yaml files may point to tqtc- prefixed
repos, which can lead to build failures when doing a top-level build
and the local repo directory names have no tqtc- prefix.
This is the case both in the CI, and when using init-repository or
git submodule init --recursive, because qt5.git specifies the 'path'
key for each submodule to not contain any tqtc- prefix.
Normalize the repo names by removing the tqtc- prefix when doing
dependency resolution for CMake add_subdirectory calls, if such a
submodule name does not exist on-disk.
The normalization is conditional, to allow inclusion of repos that
don't have a non-tqtc mirror.
qt_internal_sync_to and the other git related operations
are currently broken (both before and after this change) when
used in conjunction with tqtc- repos and is non-trivial to fix.
The first problem is the assumption of using the 'origin' remote,
which will likely be an open-source repo that doesn't contain
any tqtc repos.
The second problem is that we would need to agree upon requiring
2 remotes, one open source and one tqtc one, to reliably choose
where to clone / fetch from, as well as determining whether
the checked out repo name needs to have a tqtc- prefix
(by checking whether the repo does not exist in the open source
remote for commercial only repos).
Alternatively we could hard code a list of known open source repos,
and anything not in the list will have its tqtc- prefix kept, but we
still need to know which remote to use.
As a drive-by, adjusted some of the shown messages for better
readability and easier grepping.
Pick-to: 6.2 6.3
Fixes: QTBUG-102883
Change-Id: I6806b119dd32b14dc0d9711dc829bfc5130d1e6f
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
While resolving inter-repository dependencies, we skipped dependencies
if their dependencies.yaml did not exist. This is wrong. Instead, such
repos must be handled as if they don't have dependencies.
Drive-by change: Clarify the dependencies result value in the comment
for qt_internal_resolve_module_dependencies.
This amends commit 8a94d1e294.
Fixes: QTBUG-100549
Change-Id: I523ed74cfa96a4bab238f3a475f28c3ac55dd7b4
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
In certain cases the dependency resolution between Qt repositories
resulted in a wrong order, for example qtsvg being handled before
qtbase. See the linked task for an example.
Replace the existing algorithm with a post-order traversal of the
dependency graph to produce a topological ordering.
Rename qt_internal_add_module_dependencies to
qt_internal_resolve_module_dependencies and remove unnecessary
positional arguments. Use keyword arguments for a nicer API.
Raise the cmake_policy in QtSynchronizeRepo.cmake to avoid policy
warnings we now would get due to the use of IN_LIST.
Fixes: QTBUG-98268
Change-Id: I1425fd9c802fa71ae42549ceb14bcfc4c0a62078
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
If a subdirectory has a CMakeLists.txt file, but no dependency.yaml file
qt_internal_add_module_dependencies will bail out early, with
out_module_dependencies being empty.
Pick-to: 6.1
Fixes: QTBUG-92257
Change-Id: I024e812e9debd3c83b24bf306ff0375b2cb08659
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Read the 'required' value from dependencies.yaml and store all required
dependencies of repository 'qtfoo' in a global property
QT_REQUIRED_DEPS_FOR_qtfoo.
Check this property in the top-level CMakeLists.txt and only print
informational messages instead of errors for optional dependencies.
Fixes: QTBUG-91144
Change-Id: I0e1b84a70221857cebba1b9a27456ad3667bfe3a
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Having several toplevel worktrees for the same branch is problematic, but
but we do want to be able to have one toplevel worktree synchronized to
e.g. one submodule's dev and another with all submodules at the respective
dev heads.
So fall back to detached checkout if regular checkout fails, and generate
a warning.
Change-Id: I13981ab931cbbd8b56187979708bfe63f7373718
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Since we later check the worktree out to the revision specified
in the dependency file, add it as detached to avoid conflicts
with already existing worktrees for the default branch.
As a drive-bye, fix a typo in a comment.
Pick-to: 6.0
Change-Id: I81c30c1fa3cfc9831fe7884a103746b43478dd8e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
When fetching dependencies, the submodule directory might not exist
or might be empty. If so, try to initialize the submodule before trying
to add the worktree.
Pick-to: 6.0
Change-Id: Ie4d6ec483ac9ea3f9207d5ab4cea34d6e0fb2921
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Modern git usage for submodules is that the git object storage is absorbed
into the supermodule (see git submodule absorbgitdirs).
Handle both cases when trying to locate the gitdir for the target module.
Pick-to: 6.0
Change-Id: Icddbae3a4d9a8823551106f089dbbff2c935ff35
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Since doing either for qtbase and qtdeclarative can take a long time,
and since it's a relevant change to the local file system, log what's
going on.
Swallow output from most git commands unless cmake runs in VERBOSE
mode.
Pick-to: 6.0
Change-Id: I984915689247d6372240744b5bc9183660046084
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
For missing dependencies, create worktrees from existing clones if
found, otherwise clone from the same remote.
Change-Id: Ie624e6c1b3f621dce43ee8cc7f06ce469851285b
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Make a few modifications to the dependency evaluation to include the
revision of each dependency, and a method that checks each dependency
out to the revision necessary to create a consistent set for the
requested module.
If the requested module is ".", check all modules out to the given
revision.
Can be called (ideally from a git-sync-to alias script):
cmake -DSYNC_TO_MODULE="$1" -DSYNC_TO_BRANCH="$2" \
-P cmake/QtSynchronizeRepo.cmake
Change-Id: I007e9f9023bae949907b64e264ae7869dff1da2e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Put dependencies of all modules to top-level variables and
check for unmet dependencies before module configuring.
Fixes: QTBUG-88214
Change-Id: I089feb474687652f3f8fd1bb1959179ea1114983
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Don't rely on .gitmodules, instead parse the dependencies.yaml file from
every subdirectory with a CMakeLists.txt, and sort all projects based
on that data. Projects with no dependencies are added last.
This allows us to get rid of the duplication of dependency information
in .gitmodules, and makes each module the authoritative source of its
own dependencies.
Change-Id: Ib1ec6c63bde2aa1852399d598dac5b8e1efda31d
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Alexandru Croitor
Alexey Edelev
Joerg Bornemann
Kai Köhne
Volker Hilsheimer