2023-03-09 17:30:40 +00:00
#!/usr/bin/env bash
#
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2013-2023 Igor Pecovnik, igor@armbian.com
#
# This file is a part of the Armbian Build Framework
# https://github.com/armbian/build/
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
function cli_docker_pre_run( ) {
if [ [ " ${ DOCKERFILE_GENERATE_ONLY } " = = "yes" ] ] ; then
display_alert "Dockerfile generation only" "func cli_docker_pre_run" "debug"
return 0
fi
2023-01-30 21:48:35 +00:00
case " ${ DOCKER_SUBCMD } " in
shell)
# inside-function-function: a dynamic hook, only triggered if this CLI runs.
function add_host_dependencies__ssh_client_for_docker_shell_over_ssh( ) {
2023-03-28 18:20:56 +00:00
declare -g EXTRA_BUILD_DEPS = " ${ EXTRA_BUILD_DEPS } openssh-client "
2023-01-30 21:48:35 +00:00
}
declare -g DOCKER_PASS_SSH_AGENT = "yes" # Pass SSH agent to docker
; ;
esac
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
# make sure we're not _ALREADY_ running under docker... otherwise eternal loop?
if [ [ " ${ ARMBIAN_RUNNING_IN_CONTAINER } " = = "yes" ] ] ; then
2023-01-29 16:07:02 +00:00
exit_with_error "asking for docker... inside docker. how did this happen? Tip: you don't need 'docker' to run armbian-next inside Docker; it's automatically detected and used when appropriate."
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
fi
}
function cli_docker_run( ) {
2023-02-01 14:39:19 +00:00
# Docker won't have ${SRC}/.git, so precalculate the git-info header so it can be included in the inside-Docker logs.
# It's gonna be picked up by export_ansi_logs() and included in the final log, if it exists.
declare -g GIT_INFO_ANSI
GIT_INFO_ANSI = " $( prepare_ansi_git_info_log_header) "
2025-01-03 10:44:00 +00:00
# GIT_INFO_ANSI can grow to be quite large if there are many changed files.
# If it's too big, it will cause "argument list too long" errors when launching docker.
# Limit it to 1024 characters, otherwise replace it with a simple message.
if [ [ ${# GIT_INFO_ANSI } -gt 1024 ] ] ; then
GIT_INFO_ANSI = "Armbian: too many git changes to list."
fi
2023-02-01 14:39:19 +00:00
2023-05-24 14:32:31 +00:00
# Same stuff for BUILD_REPOSITORY_URL and BUILD_REPOSITORY_COMMIT.
if [ [ -d " ${ SRC } /.git " && " ${ CONFIG_DEFS_ONLY } " != "yes" ] ] ; then # don't waste time if only gathering config defs
set_git_build_repo_url_and_commit_vars "docker launcher"
fi
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
LOG_SECTION = "docker_cli_prepare" do_with_logging docker_cli_prepare
2023-01-29 11:41:49 +00:00
# @TODO: and can be very well said that in CI, we always want FAST_DOCKER=yes, unless we're building the Docker image itself.
if [ [ " ${ FAST_DOCKER :- "no" } " != "yes" ] ] ; then # "no, I want *slow* docker" -- no one, ever
LOG_SECTION = "docker_cli_prepare_dockerfile" do_with_logging docker_cli_prepare_dockerfile
2023-01-30 12:25:02 +00:00
if [ [ " ${ DOCKERFILE_GENERATE_ONLY } " = = "yes" ] ] ; then
display_alert "Dockerfile generated" "exiting" "info"
exit 0
fi
2023-01-29 11:41:49 +00:00
LOG_SECTION = "docker_cli_build_dockerfile" do_with_logging docker_cli_build_dockerfile
fi
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
LOG_SECTION = "docker_cli_prepare_launch" do_with_logging docker_cli_prepare_launch
ARMBIAN_CLI_RELAUNCH_PARAMS += ( [ "SET_OWNER_TO_UID" ] = " ${ EUID } " ) # fix the owner of files to our UID
ARMBIAN_CLI_RELAUNCH_PARAMS += ( [ "ARMBIAN_BUILD_UUID" ] = " ${ ARMBIAN_BUILD_UUID } " ) # pass down our uuid to the docker instance
ARMBIAN_CLI_RELAUNCH_PARAMS += ( [ "SKIP_LOG_ARCHIVE" ] = "yes" ) # launched docker instance will not cleanup logs.
2024-02-01 14:40:33 +00:00
# Produce the re-launch params.
declare -g ARMBIAN_CLI_FINAL_RELAUNCH_ARGS = ( )
declare -g ARMBIAN_CLI_FINAL_RELAUNCH_ENVS = ( )
produce_relaunch_parameters # produces ARMBIAN_CLI_FINAL_RELAUNCH_ARGS and ARMBIAN_CLI_FINAL_RELAUNCH_ENVS
# Add the relaunch envs to DOCKER_ARGS.
for env in " ${ ARMBIAN_CLI_FINAL_RELAUNCH_ENVS [@] } " ; do
display_alert "Adding Docker env" " ${ env } " "debug"
DOCKER_ARGS += ( "--env" " ${ env } " )
done
2022-11-30 10:16:17 +00:00
case " ${ DOCKER_SUBCMD } " in
shell)
display_alert "Launching Docker shell" "docker-shell" "info"
2024-07-12 20:35:29 +00:00
docker run -it " ${ DOCKER_ARGS [@] } " " ${ DOCKER_ARMBIAN_INITIAL_IMAGE_TAG } " /bin/bash
2022-11-30 10:16:17 +00:00
; ;
purge)
display_alert "Purging unused Docker volumes" "docker-purge" "info"
docker_purge_deprecated_volumes
; ;
*)
2023-01-18 20:03:13 +00:00
# this does NOT exit with the same exit code as the docker instance.
# instead, it sets the docker_exit_code variable.
declare -i docker_exit_code docker_produced_logs = 0
2023-04-20 16:36:00 +00:00
docker_cli_launch # MARK: this "re-launches"
2023-01-18 20:03:13 +00:00
# Set globals to avoid:
# 1) showing the controlling host's log; we only want to show a ref to the Docker logfile, unless it didn't produce one.
2023-01-21 01:45:24 +00:00
# If it did produce one, it's "link" is already shown above.
2023-01-18 20:03:13 +00:00
if [ [ $docker_produced_logs -gt 0 ] ] ; then
declare -g show_message_after_export = "skip" # handled by export_ansi_logs()
fi
# 2) actually exiting with the same error code as the docker instance, but without triggering an error.
declare -g -i global_final_exit_code = $docker_exit_code # handled by .... @TODO
2022-11-30 10:16:17 +00:00
; ;
2023-01-18 20:03:13 +00:00
2022-11-30 10:16:17 +00:00
esac
armbian-next: the great cli entrypoint (+docker) rewrite; introduce `USE_LOCAL_APT_DEB_CACHE` replacing `apt-cacher-ng`
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
2022-10-09 15:58:23 +00:00
}