- it's more and more common that the (bare) DT files in our `dt` folders have landed upstream
- this adds warnings and marks the patching table red when some bare-dt file overwrites what's already in git
- without this it's very easy to forget them there during bumps
* tools: shellfmt: Beautify and add board configs to formatting list
* Format the whole code using `lib/tools/shellfmt.sh`
---------
Co-authored-by: Igor <igor@armbian.com>
This fixes detection of the config option for SpacemiT K1 which is `CONFIG_SOC_SPACEMIT_K1X`.
Otherwise it would not find it because it includes a '1'.
- `git format-patch --zero-commit` doesn't affect `index xxx...yyy` lines, only `From: `
- so use the _classy_ "use a regex with a callback" solution as git format-patch doesn't offer one
- this will make _all_ patches change when rewritten, but hopefully _for the last time_ !
- we need to preserve `index 000000000000..xxx` as zeros, which indicate new file creation, thus:
- new file creations are rewritten as `index 000000000000..111111111111`
- non-creations are rewritten as `index 111111111111..222222222222`
- this is the final version of #6455
- `inventory-boards-csv.py`: moar Python atrocities done to data; idea here is to output one line per board, with <<varies>> for fields that, well, vary (mainly across BRANCH'es)
- `inventory-boards` sets `TARGETS_FILE` to something that doesn't exist, so the `default-targets.yaml` is used (so same list for everyone, save for userpatched-boards)
This fixes the readability of colorized output of compile.sh when
running on a terminal with a light background. It uses the COLORFGBG
environment variable similarly to how the ip(8) command does.
Signed-off-by: Darsey Litzenberger <dlitz@dlitz.net>
- without incremental: true:
- we replace the DT Makefile based on *.dts at the DT directory after all patching/copying is done
- overlay target is added if overlays present
- with incremental: true:
- we parse the Makefile, parse the DTs being built, and only add .dts's we have added from 'dt' directory
- overlays target are not added; we assume smth else already added them
```
# For each item in chunk, check if it is going to run in a GH-hosted runner or self-hosted, and set some matrix variables
# accordingly; self_hosted: yes/no, and more specifically, fetch_depth, which is 0 for self-hosted, and 1 for GH-hosted.
# The reasoning for this is that git clones are much faster if not shallow on self-hosted, but much slower on GH-hosted.
```
- `self_hosted` -> `shosted` for brevity in JSON matrix
- `fetch_depth` -> `fdepth` for brevity in JSON matrix
- split errors and warnings into multiple lines, so a bit easier to see in the logs what the real error was
- also turn down a few spurious warnings to debugs
- digs into config/distributions and config/desktops for info
- this produces `output/info/all_userspace_inventory.json`
- this is now passed down to the `targets-compositor` in `cli-jsoninfo`
- `targets-compositor` now accepts `userspace:` as `items-from-inventory`
- extra: add `targets-composed` CLI command, to stop after targets-compositor
- also avoid log archiving during individual download jobs via SKIP_LOG_ARCHIVE=yes
- I've tested with PARALLEL_DOWNLOADS_WORKERS=16 and it saturates my gigabit link, ghcr.io is great at reads
- more than ~16-ish might be too much though
> How to use:
>
> `./compile.sh inventory` - does just the board inventory; look for output in `output/info`
>
> `./compile.sh targets-dashboard` - does inventory, targets compositing, and images info; look for output in `output/info`, read the instructions output by the command if you want to load the OpenSearch dashboards.
>
> `./compile.sh targets` - does the full targets compositing and artifacts, look for output in `output/info`
>
> If you don't have a `userpatches/targets.yaml`, _one will be provided for you_ defaulting to Jammy minimal CLI
> and Jammy xfce desktop, for all boards in all branches. You can pass filters via `TARGETS_FILTER_INCLUDE=...` to narrow.
>
- board JSON inventory:
- more generic regex parsing of variables from board files:
- all top-level (non-indented) variables are parsed and included in the JSON board inventory
- this allows us to add new variables to the board files without having to update the parser
- variables can be bare, `export` or `declare -g`, but **_must_ be quoted** (single or double) and UPPER_CASE
- some special treatment for certain variables:
- `KERNEL_TARGET` is parsed as a _comma-separated_ list of valid BRANCH'es
- `BOARD_MAINTAINER` is parsed as _space-separated_ list of valid maintainer GH usernames as `BOARD_MAINTAINERS: [...]` in the JSON
- script complains if `BOARD_MAINTAINER` is not set in core boards. Empty is still allowed.
- `HAS_VIDEO_OUTPUT="no"` causes `BOARD_HAS_VIDEO: false` in the JSON (for desktop-only inventorying, see below)
- introduce `not-eos-with-video` in `items-from-inventory` at the targets compositor
- the same as `not-eos`, but with added `BOARD_HAS_VIDEO: true` filter, see above
- introduce `TARGETS_FILTER_INCLUDE` for targets compositor
- this filters the targets _after_ compositing (but before getting image info), based on the board inventory data
- it's a comma-separated list of `key:value` pairs, which are OR-ed together
- new virtual info `BOARD_SLASH_BRANCH` post-compositing inventory for filtering of a specific BOARD/BRANCH combo (e.g. `odroidhc4/edge`)
- some interesting possible filters:
- `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4"`: _only_ build a single board, all branches. JIRA [AR-1806]
- `TARGETS_FILTER_INCLUDE="BOARD_SLASH_BRANCH:odroidhc4/current"`: _only_ build a single board/branch combo
- `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4,BOARD:odroidn2"`: _only_ build _two_ boards, all branches.
- `TARGETS_FILTER_INCLUDE="BOARD_MAINTAINERS:rpardini"`: build all boards and branches where rpardini is a maintainer
- `TARGETS_FILTER_INCLUDE="BOARDFAMILY:rockchip64"`: build all boards and branches in the rockchip64 family
- image-info-only variables like `LINUXFAMILY` is **not** available for filtering at this stage
- rename `config/templates` `targets-all-cli.yaml` to `targets-default.yaml`
- this is used when no `userpatches/targets.yaml` is found
- new default includes all boards vs branches for non-EOS boards
- also desktop for all boards that _don't_ have `HAS_VIDEO_OUTPUT='no``
- introduce simplified `targets-dashboard` CLI:
- does only inventory, compositing, and image info, but not artifact reducing, etc.
- ignore desktop builds in the OpenSearch indexer
- update the OpenSearch Dashboards, including new information now available
- invert the logic used for `CLEAN_INFO` and `CLEAN_MATRIX`
- defaults to `yes` now, so new users/CI don't get hit by stale caches by default
- repo pipeline CLI stuff is usually run on saved/restored artifacts for `output/info`, so don't clean by default via the CLI
- even Rich'er patch output by colorizing certain strings green/yellow/red
- BASE_GIT_TAG now very sneakily also accepts a branch name
- IMPORTANT: this includes: BREAKING CHANGE: patches failing to apply now break the build. fixes#4958
- also break on legacy `process_patch_file()` failure, remove `EXIT_PATCHING_ERROR`
- this way it's consistent across all artifacts and images
- debs-to-repo-download: fix: use DEB_STORAGE instead of hardcoding output/debs (so BETA=yes works)
also:
- fix `Duplicate oci_target` message in mapper-oci-uptodate
Ricardo Pardini
igorpecovnik
amazingfate
ColorfulRhino
Darsey Litzenberger
iysheng
Igor Pecovnik