diff --git a/.automation/skills/upstream-rebase/SKILL.md b/.automation/skills/upstream-rebase/SKILL.md new file mode 100644 index 00000000000000..c23286bb72c4a9 --- /dev/null +++ b/.automation/skills/upstream-rebase/SKILL.md @@ -0,0 +1,247 @@ +--- +name: upstream-rebase +description: Nightly rebase of edera/mainline onto torvalds master, run unattended in CI by .github/workflows/upstream-rebase.yml. Covers the replay, conflict resolution, build checks, the audit of upstream fixes that meet the series, and the report the workflow publishes. +--- + +# Nightly upstream rebase + +You are rebasing the Edera downstream kernel series (around 80 commits: +Hyper-V-nested-on-Xen, the Xen PV-IOMMU, NUMA-aware Xen backends, 2 MiB +ballooning, OpenPaX, and assorted fixes) on `edera/mainline` onto Linus' +tree. The goal is a replay that changes **nothing downstream**: every +difference in the final tree must come from the upstream delta alone. Where +that is impossible, because upstream and downstream touched the same code, you +resolve the conflict the way a careful kernel maintainer would and you say +exactly what you did. + +This runs unattended. Nobody is watching while you work, and nobody can answer +a question. Everything a reviewer needs has to end up in the report. + +## What happens to your result + +You cannot push, and nothing you write can cause the branch to be pushed. +Your report can only stop it. When you finish, the workflow independently: + +1. runs `verify-upstream-rebase.sh` against your branch, which checks from git + alone that the series sits on the upstream tip, is linear, lost or gained + no commit, and changed the tree by exactly the upstream delta; +2. builds x86_64 and arm64 `vmlinux` with `kernel-build.sh`, which turns on + every downstream feature as a built-in. + +If all of that passes **and your report's "Needs a decision" section is +`None.` with nothing marked UNSURE anywhere** (checked by +`report-needs-decision.sh`), the downstream branch is force-pushed to your +branch. Otherwise your branch goes up as a pull request with your report as +the body, and a maintainer reads it. + +So a question you raise is never lost to an automatic push. Raise one whenever +a maintainer should see something before the branch moves, even when the +rebase itself is clean, for example an upstream change that reaches a +downstream feature without touching the same lines. Do not use the word UNSURE +for anything else. + +So: a conflict you resolve **will** show up as drift and **will** go to a +human. That is the intended outcome, not a failure. Do not try to make a +resolution look clean, and never drop, squash, reorder or reword a downstream +commit to get past the checks. A rebase that reaches review with an honest +report is a success; one that hides a judgement call is not. + +## Inputs + +The workflow gives you, in the prompt: + +- `DOWNSTREAM`: the branch being rebased, `edera/mainline`. +- `OLD_TIP`: its current commit (already checked out). +- `UPSTREAM`: the upstream commit to rebase onto, fetched as the local branch + `upstream-target`. +- `RESULT`: the local branch name your result must end up on. +- `REPORT`: the path your report must be written to. + +The automation's own scripts are in `.rebase-tools/scripts/`, copied from the +default branch. Use those, never a copy inside the tree you are rebasing: the +tree is what is under test. + +### How to run commands here + +Your shell commands are checked against an allowlist that matches the +command's first word. Shell variables, `$(...)`, `<(...)` and `VAR=x cmd` +prefixes will be refused, so in every snippet below `OLD_TIP`, `UPSTREAM`, +`RESULT`, `MB` and `<...>` are placeholders: substitute the literal SHA or +path, or a path you choose. Put scratch files, worktrees and build output +under `.rebase-scratch/` in the repository root (`mkdir -p .rebase-scratch`); +the kernel's `.gitignore` ignores it and nothing reads it after you. + +This is a blobless clone: file contents are fetched from GitHub the first time +a command needs them, so the first `git log -p` or checkout over a range is +slower than you expect. That is not a hang. + +## 1. Survey + +```sh +git merge-base OLD_TIP UPSTREAM # call the result MB +git rev-list --count --no-merges MB..OLD_TIP # downstream commits +git log --oneline --no-merges MB..UPSTREAM # what is new upstream +``` + +The upstream range is large (a few days of mainline, far more during a merge +window), so do not read it all. Find where it meets the +series: + +```sh +git diff --name-only --output=.rebase-scratch/down.files MB..OLD_TIP +git diff --name-only --output=.rebase-scratch/up.files MB..UPSTREAM +comm -12 .rebase-scratch/down.files .rebase-scratch/up.files # both sorted already +git log --oneline --no-merges MB..UPSTREAM -- +``` + +Read the messages and diffs of those upstream commits. They are where a +textually clean replay can still be semantically wrong: a helper downstream +calls that changed its locking or return convention, a struct field that +moved, a Kconfig symbol that was renamed. Note them; you come back to them in +step 4. + +The series may contain a merge commit from an old pull request. Rebasing +flattens it into its commits, which is expected and the checker accepts it; +do not use `--rebase-merges`. + +## 2. Rebase + +```sh +git switch -c RESULT OLD_TIP +git rebase --onto UPSTREAM MB RESULT +``` + +When a commit conflicts: + +- Read the upstream change that caused it **and** the downstream commit's + intent (its message, and the rest of its diff). Resolve so the downstream + commit does what it did before, on top of what upstream now does. +- Upstream wins on fixes. If an upstream fix changed the code a downstream + commit edits, keep every check, lock, ordering constraint and error path + the fix introduced, and fit the downstream change around it. +- If upstream now contains the downstream change itself (it was upstreamed, + perhaps through a subsystem tree), let the downstream commit go empty and + let git drop it. Record that, with the upstream commit. +- If you cannot tell what the right resolution is, do not guess silently. Make + the most conservative resolution you can defend, mark it **UNSURE** in the + report, and explain both readings. +- Never use `-X ours`, `-X theirs`, `--skip`, or `git checkout --ours/--theirs` + on a whole file to make a conflict go away. + +For each conflict, record: the downstream commit (subject), the files, the +upstream commit it collided with, and in a sentence or two what you kept and +why. + +## 3. Build + +The workflow re-runs these builds itself, but run them here so you can fix +what your resolutions broke. Each takes a while; run them once the rebase is +complete, not after every commit. + +```sh +bash .rebase-tools/scripts/kernel-build.sh x86_64 .rebase-scratch/obj-x86 4 +bash .rebase-tools/scripts/kernel-build.sh arm64 .rebase-scratch/obj-arm64 4 +``` + +Exit status 3 means a downstream option no longer survives `olddefconfig`: +upstream renamed or re-depended a Kconfig symbol the series relies on (or that +the build script names). Find out which, and say so in the report; if the +build script's list is what is stale, that is a **Needs a decision** item, +not something to work around. + +If a build fails because of the rebase (a resolution, or an upstream change a +downstream commit has not caught up with), fix it **in the downstream commit +that is wrong**. Interactive editors do not work here, so drive the todo list +with sed: `git -c sequence.editor="sed -i 's/^pick /edit /'" rebase +-i UPSTREAM`, amend, then `git rebase --continue`. Do not add a fix-up commit +on top unless there is no single commit it belongs to; if you must, its +subject starts with `rebase: ` and the report says why. + +If a build fails on the pristine upstream tree too, it is not yours: build +UPSTREAM the same way in a `.rebase-scratch/` worktree to confirm, and report +it as pre-existing. + +## 4. Audit what upstream brought in + +For every upstream commit flagged in step 1, check that its change **survived +the replay**. Ancestry is not enough: a downstream commit replayed on top can +edit the very lines a fix added. + +Give fixes the most attention: anything with a `Fixes:` tag, a `CVE-` +reference, or `Cc: stable`. For each one that touched a file the series also +touches: + +```sh +git log --oneline UPSTREAM..RESULT -- +``` + +If that returns anything, read the **final tree**, not the commit graph: +confirm the fix's checks and ordering are intact and that downstream code +added after it is covered by them. Grep tree-wide for any function or flag +the fix deliberately removed, in case a downstream commit brought it back. + +Also look for semantic collisions that did not conflict: an upstream change +to a function's contract (locking, refcounting, return values, a new required +call) where a downstream commit calls that function. Each one you cannot rule +out is an UNSURE item. + +## 5. Check yourself + +Run the same checker the workflow will run, so the report can explain its +result rather than be surprised by it: + +```sh +bash .rebase-tools/scripts/verify-upstream-rebase.sh OLD_TIP RESULT UPSTREAM +``` + +Every drift line and every changed commit it lists must be accounted for in +your report by a conflict resolution, a dropped-empty commit or a build fix. +If one is not, you made a change you did not mean to: find it and undo it. + +## 6. Report + +Write the report to the `REPORT` path, in Markdown. It becomes a pull request +body when a human has to look, so lead with what they must decide. No +preamble, no sign-off. + +```markdown +## Summary +One paragraph: the tree, upstream range (N commits, short SHAs and the +`git describe` of each end), downstream commits replayed, conflicts resolved, +commits dropped, and whether you expect the checker to pass. + +## Needs a decision +One bullet per question, each starting **UNSURE:**, with both readings and +what you committed. "None." if there is nothing; anything else here stops the +automatic push. + +## Conflicts resolved +Per conflict: downstream commit, files, colliding upstream commit, what you +kept and why. "None." if none. + +## Dropped downstream commits +Subject and the upstream commit that made it empty. "None." if none. + +## Build fixes +What broke, which commit you fixed it in, why. "None." if none. + +## Upstream commits that meet the series +`short-sha subject` for each upstream commit that touched a file the series +touches, and for each fix among them your reading of the final tree. "None." +if none. + +## Builds +x86_64, arm64: pass, fail (with the first error), or pre-existing failure +(confirmed on pristine upstream). +``` + +Then stop. Leave RESULT checked out or not; the workflow reads the branch, +not the working tree. + +## If you cannot finish + +If the rebase cannot be completed (a conflict you cannot resolve at all, a +toolchain that will not run), run `git rebase --abort`, do not create the +RESULT branch, and write the report explaining where you stopped and why. The +workflow treats a missing RESULT branch as a failed night and opens an issue +with your report. diff --git a/.github/scripts/kernel-build.sh b/.github/scripts/kernel-build.sh new file mode 100755 index 00000000000000..2a4c38dd68cb80 --- /dev/null +++ b/.github/scripts/kernel-build.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Builds the kernel the way the nightly automation judges a branch. +# +# defconfig alone compiles almost none of the Edera series: Hyper-V nesting, +# Xen dom0 and backends, the PV-IOMMU, OpenPaX and the NUMA-aware Xen paths +# are all off in it. This turns them on as built-ins and builds vmlinux, so a +# replay that breaks any of them, or links against a symbol upstream removed, +# fails here. Building only a subdirectory would miss those link errors. +# +# The rebase skill runs this same script before they finish, so +# what Claude checks and what the workflow checks cannot drift apart. +# +# Usage: +# kernel-build.sh [jobs] +# +# Run from the top of a kernel tree. is created if missing and may be +# reused between runs. For arm64 it uses CROSS_COMPILE (default +# aarch64-linux-gnu-) unless LLVM is set, in which case it builds with clang. +# +# Exit status: make's on a build failure; 3 if an option it asked for did not +# survive olddefconfig (a Kconfig dependency the series no longer satisfies). + +set -euo pipefail + +if [ $# -lt 2 ] || [ $# -gt 3 ]; then + echo "usage: $0 [jobs]" >&2 + exit 2 +fi +arch=$1 +objdir=$2 +jobs=${3:-$(nproc)} + +common=( + XEN XEN_BACKEND XEN_DOM0 XEN_BALLOON XEN_GNTDEV XEN_GRANT_DEV_ALLOC + NUMA ACPI_NUMA MEMORY_HOTPLUG MEMORY_HOTREMOVE ZONE_DEVICE + XEN_UNPOPULATED_ALLOC XEN_BACKEND_NUMA_AFFINITY + XEN_NETDEV_FRONTEND XEN_NETDEV_BACKEND XEN_BLKDEV_FRONTEND XEN_BLKDEV_BACKEND + NET_9P NET_9P_XEN 9P_FS + DRM DRM_VIRTIO_GPU FW_CFG_SYSFS + SECURITY OPENPAX OPENPAX_SOFTMODE OPENPAX_XATTR_PAX_FLAGS OPENPAX_MPROTECT + OPENPAX_EMUTRAMP +) +case $arch in +x86_64) + karch=x86 + # make reads LLVM from the environment; x86 is always the gcc build. + unset LLVM + cross=() + opts=( + "${common[@]}" + HYPERVISOR_GUEST PARAVIRT XEN_PV XEN_PVHVM_GUEST PCI_XEN + XEN_PCIDEV_FRONTEND XEN_PCIDEV_BACKEND XEN_IOMMU + HYPERV HYPERV_VMBUS HYPERV_NET PCI_HYPERV PCI_HYPERV_INTERFACE + ) + ;; +arm64) + karch=arm64 + if [ -n "${LLVM:-}" ]; then + cross=(LLVM=1) + else + cross=(CROSS_COMPILE="${CROSS_COMPILE:-aarch64-linux-gnu-}") + fi + opts=("${common[@]}") + ;; +*) + echo "unknown arch: $arch" >&2 + exit 2 + ;; +esac + +mkdir -p "$objdir" +mk() { make -s ARCH="$karch" "${cross[@]}" O="$objdir" "$@"; } + +mk defconfig +enable=() +for o in "${opts[@]}"; do enable+=(--enable "$o"); done +scripts/config --file "$objdir/.config" "${enable[@]}" +mk olddefconfig + +# scripts/config writes whatever it is told; olddefconfig then quietly drops +# anything whose dependencies are not met. Catch that, or a Kconfig change +# could switch a downstream feature off without failing the build. +lost=() +for o in "${opts[@]}"; do + grep -qx "CONFIG_$o=y" "$objdir/.config" || lost+=("$o") +done +if [ "${#lost[@]}" -ne 0 ]; then + echo "kernel-build: not enabled after olddefconfig ($arch): ${lost[*]}" >&2 + exit 3 +fi + +mk -j"$jobs" vmlinux +echo "kernel-build: $arch vmlinux built in $objdir" diff --git a/.github/scripts/report-needs-decision.sh b/.github/scripts/report-needs-decision.sh new file mode 100755 index 00000000000000..e15b51979fd3b7 --- /dev/null +++ b/.github/scripts/report-needs-decision.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Decides whether the nightly rebase report asks a human for a decision. +# +# The upstream-rebase skill tells Claude to list anything it was unsure about +# under a "## Needs a decision" section, each item marked UNSURE. A rebase can +# be clean by every mechanical measure (no drift, builds pass) and still raise +# such a question: an upstream change that interacts with a downstream feature +# without touching the same lines. The workflow does not push those +# automatically; it opens a pull request so a maintainer sees the question +# before the branch moves. +# +# The report counts as asking for a decision when either: +# - its "Needs a decision" section has any content other than "None.", or +# - any line marks something UNSURE. +# Both err towards a pull request: a stray "UNSURE" costs one review, a missed +# question costs an unreviewed kernel change. +# +# Usage: +# report-needs-decision.sh +# +# Prints the section's content when it asks for a decision. Exit status: +# 0 a decision is needed +# 1 no decision is needed +# 2 the report cannot be read + +set -euo pipefail + +if [ $# -ne 1 ] || [ ! -r "$1" ]; then + echo "usage: $0 " >&2 + exit 2 +fi + +awk ' + BEGIN { insec = 0; body = ""; unsure = 0 } + /UNSURE/ { unsure = 1 } + /^##?[ \t]/ { + if (tolower($0) ~ /^##?[ \t]+needs a decision[ \t]*$/) { insec = 1; next } + insec = 0 + } + insec { + line = $0 + gsub(/^[ \t]+|[ \t]+$/, "", line) + if (line == "") next + if (tolower(line) ~ /^(_?none\.?_?|n\/a\.?)$/) next + body = body $0 "\n" + } + END { + if (body != "" || unsure) { printf "%s", body; exit 0 } + exit 1 + } +' "$1" diff --git a/.github/scripts/tests/report-needs-decision.test.sh b/.github/scripts/tests/report-needs-decision.test.sh new file mode 100755 index 00000000000000..b2ab3547a2f1b6 --- /dev/null +++ b/.github/scripts/tests/report-needs-decision.test.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Tests for .github/scripts/report-needs-decision.sh. +# +# A clean nightly rebase is force-pushed to its downstream branch unless this script says +# Claude's report asks for a decision, so a report that does ask must never +# read as one that does not. +# +# Run from anywhere: bash .github/scripts/tests/report-needs-decision.test.sh +set -uo pipefail + +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SCRIPT="$HERE/../report-needs-decision.sh" + +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +failures=0 +expect() { + # expect (report on stdin) + local desc=$1 want=$2 got + cat >"$WORK/report.md" + bash "$SCRIPT" "$WORK/report.md" >/dev/null 2>&1 + got=$? + if [ "$got" -eq "$want" ]; then + echo "ok $desc" + else + echo "FAIL $desc: exit $got, want $want" >&2 + failures=$((failures + 1)) + fi +} + +expect "no section, nothing unsure" 1 <<'EOF' +## Summary +All 221 commits replayed cleanly. + +## Conflicts resolved +None. +EOF + +expect "section says None." 1 <<'EOF' +## Summary +Clean. + +## Needs a decision +None. + +## Conflicts resolved +None. +EOF + +expect "section empty before the next heading" 1 <<'EOF' +## Needs a decision + +## Conflicts resolved +None. +EOF + +expect "section with an item" 0 <<'EOF' +## Summary +Clean replay. + +## Needs a decision +- Upstream 0822a2e890 now reaches Arm dom0 vPCI; see below. + +## Conflicts resolved +None. +EOF + +expect "UNSURE outside the section" 0 <<'EOF' +## Summary +Clean replay. + +## Conflicts resolved +- foo.c: kept downstream's lock order. **UNSURE** whether upstream intended otherwise. +EOF + +expect "heading case and trailing space" 0 <<'EOF' +## NEEDS A DECISION +- something +EOF + +expect "a level-3 heading stays inside the section" 0 <<'EOF' +## Needs a decision +### Arm vPCI +- something +## Builds +pass +EOF + +expect "a report with one UNSURE item asks for a decision" 0 <<'EOF' +## Summary +Rebased cleanly. + +## Needs a decision +- **UNSURE: upstream `0123456789ab` ("xen/events: rework lateeoi handling") now reaches the downstream _on_node bind helpers.** + +## Conflicts resolved +None. +EOF + +bash "$SCRIPT" "$WORK/missing.md" >/dev/null 2>&1 +if [ $? -eq 2 ]; then echo "ok unreadable report is an error"; else + echo "FAIL unreadable report is an error" >&2 + failures=$((failures + 1)) +fi + +if [ "$failures" -ne 0 ]; then + echo "$failures failure(s)" >&2 + exit 1 +fi +echo "all passed" diff --git a/.github/scripts/tests/verify-upstream-rebase.test.sh b/.github/scripts/tests/verify-upstream-rebase.test.sh new file mode 100755 index 00000000000000..a28d96d3ddb624 --- /dev/null +++ b/.github/scripts/tests/verify-upstream-rebase.test.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# Tests for .github/scripts/verify-upstream-rebase.sh. +# +# Builds a small throwaway repository with an upstream line and a downstream +# series on it, rebases the series onto a newer upstream, and then damages +# copies of that result in each way the checker exists to catch. The nightly +# rebase force-pushes edera/mainline on this script's say-so, so every +# damaged copy must be refused, and every honest replay must pass. +# +# Run from anywhere: bash .github/scripts/tests/verify-upstream-rebase.test.sh +set -uo pipefail + +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SCRIPT="$HERE/../verify-upstream-rebase.sh" + +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT +cd "$WORK" || exit 1 + +failures=0 +expect() { + # expect [] + local desc=$1 want=$2 got out + out=$(bash "$SCRIPT" "$3" "$4" "$5") + got=$? + if [ "$got" -ne "$want" ]; then + echo "FAIL $desc: exit $got, want $want" >&2 + printf ' %s\n' "${out//$'\n'/$'\n' }" >&2 + failures=$((failures + 1)) + elif [ $# -ge 6 ] && ! grep -qF -- "$6" <<<"$out"; then + echo "FAIL $desc: output lacks '$6'" >&2 + printf ' %s\n' "${out//$'\n'/$'\n' }" >&2 + failures=$((failures + 1)) + else + echo "ok $desc" + fi +} + +git init -q -b upstream . +git config user.name test +git config user.email test@example.invalid +commit() { git add -A && git commit -qm "$1"; } + +# Upstream: two files with room for both sides to edit. +seq 1 40 >a.c +seq 1 40 >b.c +commit "upstream: base" +base=$(git rev-parse HEAD) + +# Downstream series. Two commits share a subject, as real series do. +git switch -q -c downstream +sed -i 's/^10$/10 downstream/' a.c +commit "downstream: edit a" +sed -i 's/^30$/30 downstream/' b.c +commit "apply dep patch" +echo "new" >c.c +commit "downstream: add c" +sed -i 's/^35$/35 downstream/' b.c +commit "apply dep patch" +old=$(git rev-parse HEAD) + +# Upstream moves on in two steps. The first edits only lines far from +# anything downstream touches; the second edits line 12, which sits inside the +# hunk context of "downstream: edit a" (line 10) without conflicting with it. +git switch -q upstream +sed -i 's/^1$/1 upstream/' b.c +commit "upstream: edit b far away" +far=$(git rev-parse HEAD) +sed -i 's/^12$/12 upstream/' a.c +commit "upstream: edit a near downstream" +up=$(git rev-parse HEAD) + +replay() { # replay + git switch -q -c "$1" "$old" && git rebase -q --onto "$2" "$base" "$1" +} + +replay far "$far" +expect "replay with untouched context passes" 0 "$old" "$(git rev-parse HEAD)" "$far" "identical patch-ids" +expect "a no-op rebase passes" 0 "$old" "$old" "$base" "identical patch-ids" + +# Context moved but the commit's own lines did not: still clean. +replay rebased "$up" +good=$(git rev-parse HEAD) +expect "replay with moved context passes" 0 "$old" "$good" "$up" "1 changed only in hunk context" + +# Damage 1: one of the two same-subject commits edited. +git switch -q -c tamper "$good" +# shellcheck disable=SC2016 # expanded by the shell git rebase starts +git rebase -q --exec 'if [ "$(git log -1 --format=%s)" = "apply dep patch" ] && grep -q "35 downstream" b.c; then echo extra >>b.c; git commit -qa --amend --no-edit; fi' "$up" +expect "edited commit with a repeated subject fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "1 commit(s) lost, added, or with different +/- lines" + +# Damage 2: a commit dropped. +git switch -q -c dropped "$good" +drop=$(git log --format='%H %s' "$up..HEAD" | awk '/downstream: add c/ { print $1 }') +git rebase -q --onto "$drop^" "$drop" dropped +expect "dropped commit fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "downstream: add c" + +# Damage 3: a commit added. +git switch -q -c added "$good" +echo "sneaky" >d.c +commit "innocent looking" +expect "added commit fails" 1 "$old" "$(git rev-parse HEAD)" "$up" "innocent looking" + +# Damage 4: not on the upstream tip. +expect "stale base is refused" 2 "$old" "$old" "$up" "not based on the upstream tip" + +# Damage 5: a merge commit in the series. +git switch -q -c merged "$good" +git switch -q -c side "$up" +echo side >e.c +commit "side" +git switch -q merged +git merge -q --no-edit side +expect "merge in the series is refused" 2 "$old" "$(git rev-parse HEAD)" "$up" "merge commit" + +# A series that merged a feature branch, as both kernel trees once did: the +# first rebase flattens it, and that must still pass, since no commit is +# lost and the tree moves by exactly the upstream delta. +git switch -q -c withmerge "$old" +git switch -q -c feature "$old~1" +sed -i 's/^20$/20 feature/' a.c +commit "feature: edit a" +git switch -q withmerge +git merge -q --no-ff --no-edit feature >/dev/null +sed -i 's/^25$/25 after merge/' b.c +commit "downstream: after the merge" +mold=$(git rev-parse HEAD) +git switch -q -c flattened "$mold" +git rebase -q --onto "$up" "$base" flattened +expect "flattening a merged series passes" 0 "$mold" "$(git rev-parse HEAD)" "$up" "All 6 downstream commits replayed" + +if [ "$failures" -ne 0 ]; then + echo "$failures failure(s)" >&2 + exit 1 +fi +echo "all passed" diff --git a/.github/scripts/verify-upstream-rebase.sh b/.github/scripts/verify-upstream-rebase.sh new file mode 100755 index 00000000000000..7a493aad81c0c8 --- /dev/null +++ b/.github/scripts/verify-upstream-rebase.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# Checks a rebased downstream branch against the branch it replaces, without +# trusting whoever did the rebase. +# +# The nightly upstream-rebase workflow lets a model drive the rebase, but +# nothing it says about its own result is taken on faith. This script is the +# gate. It proves, from git alone, that: +# +# 1. the new branch sits directly on the upstream tip it claims to; +# 2. the downstream series is linear (no merge commits slipped in); +# 3. no downstream commit was lost, added, or changed: patch-ids match, or, +# where they do not, the commit's own +/- lines are identical and only a +# hunk's context anchor moved; +# 4. the tree changed by exactly the upstream delta and nothing else (only +# the +/- lines are compared, so moved hunks and blob hashes do not raise +# false alarms). +# +# A rebase that needed conflict resolution will normally fail check 4: the +# resolution is, by definition, a change to downstream lines. That is the +# point. Such a result is not wrong, but it is not "clean", and a human has to +# read it before it replaces the branch. +# +# Usage: +# verify-upstream-rebase.sh +# +# Writes a Markdown report to stdout. Exit status: +# 0 clean: all four checks pass +# 1 drift: the branch is well-formed (1-2) but 3 or 4 found differences +# 2 broken: 1 or 2 failed, or the inputs are unusable + +set -euo pipefail + +if [ $# -ne 3 ]; then + echo "usage: $0 " >&2 + exit 2 +fi + +old=$(git rev-parse --verify "$1^{commit}") +new=$(git rev-parse --verify "$2^{commit}") +up=$(git rev-parse --verify "$3^{commit}") +old_base=$(git merge-base "$old" "$up") + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +status=0 +fail() { [ "$status" -ge "$1" ] || status=$1; } +short() { git rev-parse --short=12 "$1"; } +changed_lines() { grep -E '^[+-]' | grep -vE '^(\+\+\+|---)' || true; } +changed_files() { grep -E '^(\+\+\+|---) ' || true; } + +echo "| | |" +echo "| --- | --- |" +echo "| Old tip | \`$(short "$old")\` |" +echo "| New tip | \`$(short "$new")\` |" +echo "| Old upstream base | \`$(short "$old_base")\` |" +echo "| New upstream base | \`$(short "$up")\` |" +echo + +# 1. Base. +if [ "$(git merge-base "$new" "$up")" = "$up" ]; then + echo "- [x] New branch is based on the upstream tip." +else + echo "- [ ] **New branch is not based on the upstream tip** \`$(short "$up")\`." + fail 2 +fi + +# 2. Linearity. +merges=$(git rev-list --merges "$up..$new" | wc -l) +if [ "$merges" -eq 0 ]; then + echo "- [x] Downstream series is linear." +else + echo "- [ ] **Downstream series contains $merges merge commit(s).**" + fail 2 +fi + +if [ "$status" -ge 2 ]; then + echo + echo "Stopped: the remaining checks assume a linear series on the upstream tip." + exit "$status" +fi + +# 3. Commit set. +# +# Each commit is keyed by its subject and how many earlier commits in the +# series share that subject, so the k-th "apply dep patch" before pairs with +# the k-th after. A rebase keeps the series order, so this pairs every commit +# with its own replay even when subjects repeat. +# +# A changed patch-id is fine when the commit's own +/- lines are unchanged and +# only a hunk's context anchor moved, which is what happens wherever upstream +# edited the same region. Anything else is a real difference. +series() { + git log --no-merges --reverse --format='commit %H' --patch "$1" | + git patch-id --stable | awk '{ print $2 "\t" $1 }' >"$work/pid.map" + git log --no-merges --reverse --format='%H%x09%s' "$1" | + awk -F'\t' 'NR == FNR { pid[$1] = $2; next } + { k = $2 "#" ++seen[$2]; print k "\t" $1 "\t" ($1 in pid ? pid[$1] : "-") }' \ + "$work/pid.map" - +} +series "$old_base..$old" >"$work/old.series" +series "$up..$new" >"$work/new.series" +n_old=$(wc -l <"$work/old.series") +n_new=$(wc -l <"$work/new.series") + +# key -> "old-commit new-commit verdict"; verdict is same, check, lost, added. +awk -F'\t' 'NR == FNR { oc[$1] = $2; op[$1] = $3; order[++n] = $1; next } + { + if ($1 in oc) { print oc[$1], $2, (op[$1] == $3 ? "same" : "check"); delete oc[$1] } + else print "-", $2, "added" + } + END { for (i = 1; i <= n; i++) if (order[i] in oc) print oc[order[i]], "-", "lost" }' \ + "$work/old.series" "$work/new.series" >"$work/pairs" + +own_change() { + local patch + patch=$(git show --format= "$1") + changed_files <<<"$patch" + changed_lines <<<"$patch" +} +: >"$work/moved" +: >"$work/changed" +while read -r c d verdict; do + case $verdict in + same) ;; + check) + if [ "$(own_change "$c")" = "$(own_change "$d")" ]; then + echo "$c $d" >>"$work/moved" + else + echo "$c $d" >>"$work/changed" + fi ;; + *) echo "$c $d" >>"$work/changed" ;; + esac +done <"$work/pairs" + +if [ "$n_old" -eq "$n_new" ] && [ ! -s "$work/changed" ]; then + if [ -s "$work/moved" ]; then + echo "- [x] All $n_new downstream commits replayed;" \ + "$(wc -l <"$work/moved") changed only in hunk context (listed below)." + else + echo "- [x] All $n_new downstream commits replayed with identical patch-ids." + fi +else + echo "- [ ] **Downstream commit set changed:** $n_old before, $n_new after;" \ + "$(wc -l <"$work/changed") commit(s) lost, added, or with different +/- lines." + fail 1 +fi + +# 4. Drift. +git diff "$old..$new" >"$work/rebase.diff" +git diff "$old_base..$up" >"$work/upstream.diff" +changed_files <"$work/rebase.diff" >"$work/rebase.files" +changed_files <"$work/upstream.diff" >"$work/upstream.files" +changed_lines <"$work/rebase.diff" >"$work/rebase.lines" +changed_lines <"$work/upstream.diff" >"$work/upstream.lines" + +if cmp -s "$work/rebase.files" "$work/upstream.files" && + cmp -s "$work/rebase.lines" "$work/upstream.lines"; then + echo "- [x] Zero downstream drift: the tree changed by exactly the upstream delta." +else + echo "- [ ] **Downstream drift:** the tree changed by more than the upstream delta." + fail 1 +fi + +# Details for anything that did not match. +oneline() { if [ "$1" = - ]; then echo "(none)"; else git log -1 --format='%h %s' "$1"; fi; } +if [ -s "$work/moved" ]; then + echo + echo "
Commits whose hunk context moved (own +/- lines identical)" + echo + while read -r c d; do + echo "- \`$(oneline "$c")\` → \`$(git rev-parse --short "$d")\`" + done <"$work/moved" + echo + echo "
" +fi +if [ -s "$work/changed" ]; then + echo + echo "
Commits lost, added, or whose own +/- lines changed" + echo + echo "| Before | After |" + echo "| --- | --- |" + while read -r c d; do + echo "| \`$(oneline "$c")\` | \`$(oneline "$d")\` |" + done <"$work/changed" + echo + echo "
" +fi + +if ! cmp -s "$work/rebase.lines" "$work/upstream.lines" || + ! cmp -s "$work/rebase.files" "$work/upstream.files"; then + echo + echo "
Drift (changed lines: upstream delta vs. actual change)" + echo + echo '```diff' + { + diff -u --label upstream-files --label actual-files \ + "$work/upstream.files" "$work/rebase.files" || true + diff -u --label upstream-lines --label actual-lines \ + "$work/upstream.lines" "$work/rebase.lines" || true + } | head -n 400 + echo '```' + echo + echo "
" +fi + +exit "$status" diff --git a/.github/workflows/automation-selftest.yml b/.github/workflows/automation-selftest.yml new file mode 100644 index 00000000000000..fc7305654ec509 --- /dev/null +++ b/.github/workflows/automation-selftest.yml @@ -0,0 +1,60 @@ +name: Automation self-test + +# Tests the scripts the nightly rebase relies on to decide whether to +# force-push edera/mainline, so a change that weakens either must not land +# quietly: +# +# - verify-upstream-rebase.sh: honest replays pass, including one that +# flattens a merged series; edited, dropped, added and merged commits and +# a stale base are all refused; +# - report-needs-decision.sh: any question in Claude's report, or anything +# marked UNSURE, stops the automatic push. +# +# It runs only when the automation itself changes. + +on: + pull_request: + types: [opened, synchronize, ready_for_review] + branches: ['edera/**'] + paths: + - '.github/scripts/**' + - '.github/workflows/upstream-rebase.yml' + - '.github/workflows/kernel-nightly.yml' + - '.github/workflows/rebase-land.yml' + - '.github/workflows/automation-selftest.yml' + - '.automation/**' + +permissions: + contents: read + +concurrency: + group: automation-selftest-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + selftest: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + # Only the automation is needed, not the kernel. + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + sparse-checkout: | + .github + .automation + + - name: Rebase checker tests + run: bash .github/scripts/tests/verify-upstream-rebase.test.sh + + - name: Decision detector tests + run: bash .github/scripts/tests/report-needs-decision.test.sh + + - name: Shellcheck + run: shellcheck .github/scripts/*.sh .github/scripts/tests/*.sh diff --git a/.github/workflows/kernel-nightly.yml b/.github/workflows/kernel-nightly.yml new file mode 100644 index 00000000000000..09be7062f90f45 --- /dev/null +++ b/.github/workflows/kernel-nightly.yml @@ -0,0 +1,58 @@ +name: Nightly kernel maintenance + +# Rebases edera/mainline onto torvalds master every night +# (upstream-rebase.yml). +# +# Claude does the rebase. Independent checks decide whether the result is +# pushed or proposed as a pull request; see upstream-rebase.yml. +# +# GitHub runs scheduled workflows only from the default branch, which is +# edera/6.18-lts. Until this file is there too, the schedule does not fire. +# +# Repository configuration this needs: +# vars.REBASE_APP_CLIENT_ID, secrets.REBASE_APP_PRIVATE_KEY +# A GitHub App installed on this repository with contents, pull requests +# and workflows write, allowed to bypass the protection rules on +# edera/mainline. GITHUB_TOKEN cannot do this job: it cannot push +# commits that touch .github/workflows/ (every rebase rewrites the ones +# that add them), and pull requests it opens do not trigger other +# workflows. +# vars.REBASE_FEDERATION_RULE_ID +# The workload identity federation rule for the rebase. It must accept +# this workflow's OIDC subject (a scheduled or dispatched run). +# vars.ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, +# ANTHROPIC_WORKSPACE_ID +# The rest of the Claude API federation identity. + +on: + schedule: + - cron: '23 3 * * *' # 03:23 UTC nightly + workflow_dispatch: + inputs: + force: + description: Rebase even if an open rebase pull request already covers this upstream tip + type: boolean + default: false + +permissions: + contents: read + +concurrency: + group: kernel-nightly + cancel-in-progress: false + +jobs: + mainline: + uses: ./.github/workflows/upstream-rebase.yml + permissions: + contents: read + pull-requests: read + issues: write + id-token: write + with: + downstream: edera/mainline + key: mainline + upstream_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git + upstream_branch: master + force: ${{ inputs.force == true }} + secrets: inherit diff --git a/.github/workflows/lts-backport-trigger.yml b/.github/workflows/lts-backport-trigger.yml new file mode 100644 index 00000000000000..f77e0ed1e11771 --- /dev/null +++ b/.github/workflows/lts-backport-trigger.yml @@ -0,0 +1,43 @@ +name: Trigger edera/6.18-lts backport + +# Starts edera/6.18-lts's backport of a pull request that just merged into +# edera/mainline, and does nothing else. +# +# edera/6.18-lts drives its own backports with its own copy of +# kernel-nightly.yml (backport_pr= runs lts-backport.yml there). Pull +# requests labelled no-backport, and the nightly's own rebase pull requests, +# are not dispatched; edera/6.18-lts decides everything else, including +# whether there is anything to backport at all. +# +# pull_request_target runs this file as it is on edera/mainline, with a token +# that can dispatch even for a pull request from a fork. Nothing here checks +# out or runs the pull request's code. + +on: + pull_request_target: + types: [closed] + branches: [edera/mainline] + +permissions: {} + +jobs: + dispatch: + if: >- + github.event.pull_request.merged + && !startsWith(github.event.pull_request.head.ref, 'rebase/') + && !contains(github.event.pull_request.labels.*.name, 'no-backport') + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + actions: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Dispatch + env: + GH_TOKEN: ${{ github.token }} + PR: ${{ github.event.pull_request.number }} + run: gh workflow run kernel-nightly.yml --repo "$GITHUB_REPOSITORY" --ref edera/6.18-lts -f backport_pr="$PR" diff --git a/.github/workflows/lts-rebase-trigger.yml b/.github/workflows/lts-rebase-trigger.yml new file mode 100644 index 00000000000000..7bc4e64f5c5a2c --- /dev/null +++ b/.github/workflows/lts-rebase-trigger.yml @@ -0,0 +1,35 @@ +name: Trigger edera/6.18-lts rebase + +# Starts edera/6.18-lts's own nightly rebase, and does nothing else. +# +# edera/6.18-lts drives its own rebase with its own copy of +# kernel-nightly.yml, but GitHub runs scheduled workflows only from the +# default branch, edera/mainline. So this schedule lives here and dispatches +# that copy: a workflow is identified by its path, and --ref picks the +# branch whose file runs. +# +# GITHUB_TOKEN can do this: workflow_dispatch is one of the events it is +# allowed to trigger runs with. + +on: + schedule: + - cron: '23 3 * * *' # 03:23 UTC nightly + +permissions: {} + +jobs: + dispatch: + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + actions: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Dispatch + env: + GH_TOKEN: ${{ github.token }} + run: gh workflow run kernel-nightly.yml --repo "$GITHUB_REPOSITORY" --ref edera/6.18-lts diff --git a/.github/workflows/rebase-land.yml b/.github/workflows/rebase-land.yml new file mode 100644 index 00000000000000..ce3e39ff627cd1 --- /dev/null +++ b/.github/workflows/rebase-land.yml @@ -0,0 +1,129 @@ +name: Land upstream rebase + +# Lands a nightly rebase that needed review (see upstream-rebase.yml) once a +# maintainer approves its pull request. +# +# A rebase replaces the tree rather than merging into it, so the merge button +# is the wrong tool: it would graft a second copy of the whole series onto the +# old one. Approving instead force-pushes the tree to the exact commit that +# was approved, and GitHub then marks the pull request merged. +# +# A review runs this workflow from the pull request's merge ref, which is why +# it lives on edera/mainline. +# +# What has to hold before anything is pushed: +# - the pull request is a rebase/edera-mainline/... branch from this +# repository, targeting edera/mainline; +# - the review approved the current head, not an earlier one; +# - the approver has write access; +# - the tree is still the tip the branch was made from. The branch name +# records that tip, and branch names cannot be edited on an open pull +# request, so this is not something the pull request body can change. If +# the tree moved, nothing is pushed; the next nightly run starts from the +# new tip. + +on: + pull_request_review: + types: [submitted] + +permissions: + contents: read + +concurrency: + # Shared with the nightly's publish jobs, so nothing else moves the tree + # while this runs. + group: push-${{ github.event.pull_request.base.ref }} + cancel-in-progress: false + +jobs: + land: + if: >- + github.event.review.state == 'approved' + && github.event.pull_request.head.repo.full_name == github.repository + && github.event.pull_request.base.ref == 'edera/mainline' + && startsWith(github.event.pull_request.head.ref, 'rebase/edera-mainline/') + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + pull-requests: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: App token + id: app + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.REBASE_APP_CLIENT_ID }} + private-key: ${{ secrets.REBASE_APP_PRIVATE_KEY }} + permission-contents: write + permission-pull-requests: write + permission-workflows: write + + - name: Land + env: + APP_TOKEN: ${{ steps.app.outputs.token }} + JOB_TOKEN: ${{ github.token }} + PR: ${{ github.event.pull_request.number }} + BASE: ${{ github.event.pull_request.base.ref }} + HEAD_REF: ${{ github.event.pull_request.head.ref }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + REVIEWED_SHA: ${{ github.event.review.commit_id }} + APPROVER: ${{ github.event.review.user.login }} + run: | + set -euo pipefail + say() { + GH_TOKEN=$JOB_TOKEN gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body "$1" + } + + if [ "$REVIEWED_SHA" != "$HEAD_SHA" ]; then + say "Not landed: the approval was for \`${REVIEWED_SHA:0:12}\`, but the head is now \`${HEAD_SHA:0:12}\`. Approve the current head to land it." + exit 1 + fi + + perm=$(GH_TOKEN=$JOB_TOKEN gh api "repos/$GITHUB_REPOSITORY/collaborators/$APPROVER/permission" -q .permission) + case $perm in + admin|maintain|write) ;; + *) + say "Not landed: @$APPROVER does not have write access, so their approval cannot land a rebase." + exit 1 ;; + esac + + # Name: rebase//--. + rest=${HEAD_REF#"rebase/${BASE//\//-}/"} + IFS=- read -r _date old12 up12 extra <<<"$rest" + if [ -n "${extra:-}" ] || ! [[ $old12 =~ ^[0-9a-f]{12}$ && $up12 =~ ^[0-9a-f]{12}$ ]]; then + say "Not landed: \`$HEAD_REF\` is not a branch the nightly automation made." + exit 1 + fi + + origin="https://x-access-token:${APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + cur=$(git ls-remote "$origin" "refs/heads/$BASE" | cut -f1) + if [ "${cur:0:12}" != "$old12" ]; then + say "Not landed: \`$BASE\` has moved to \`${cur:0:12}\` since this branch was made from \`$old12\`. Tonight's run starts from the new tip; this pull request will be closed as superseded." + exit 1 + fi + + # Both commits are already on GitHub, so this needs neither the + # kernel's history nor its files, only the two commit objects to + # name in the push. + git init -q work && cd work + git fetch -q --no-tags --depth=1 --filter=tree:0 "$origin" "$HEAD_SHA" "$cur" + + # Created through the API, and before the tree moves: a push from + # this shallow clone is refused once the old tip is unreferenced. + backup="backup/${BASE//\//-}-pre-rebase-$(date -u +%Y%m%d)-$old12" + GH_TOKEN=$APP_TOKEN gh api -X POST "repos/$GITHUB_REPOSITORY/git/refs" \ + -f ref="refs/heads/$backup" -f sha="$cur" >/dev/null + + git push --force-with-lease="refs/heads/$BASE:$cur" "$origin" "$HEAD_SHA:refs/heads/$BASE" + now=$(git ls-remote "$origin" "refs/heads/$BASE" | cut -f1) + if [ "$now" != "$HEAD_SHA" ]; then + say "Push reported success but \`$BASE\` is \`${now:0:12}\`, not \`${HEAD_SHA:0:12}\`. Check the run log." + exit 1 + fi + say "Landed by @$APPROVER's approval: \`$BASE\` is now \`${HEAD_SHA:0:12}\` (was \`${cur:0:12}\`, kept as \`$backup\`)." + git push "$origin" --delete "$HEAD_REF" || true diff --git a/.github/workflows/upstream-rebase.yml b/.github/workflows/upstream-rebase.yml new file mode 100644 index 00000000000000..75056019338fdf --- /dev/null +++ b/.github/workflows/upstream-rebase.yml @@ -0,0 +1,665 @@ +name: Upstream rebase + +# Rebases an Edera kernel integration tree onto its upstream. Called nightly +# by kernel-nightly.yml for edera/mainline, onto torvalds master. +# +# Claude does the rebase, following .automation/skills/upstream-rebase/ +# SKILL.md: it replays the series, resolves conflicts, fixes what the replay +# broke, builds, audits upstream fixes that meet the series, and writes a +# report. +# +# Nothing Claude says about its own result decides what happens to it. The +# model runs in a job that holds a read-only token and cannot push. Its branch +# leaves that job as a git bundle, and two things then judge it +# independently, with scripts taken from the commit this run started from +# (the default branch), never from the tree under test: +# +# - verify-upstream-rebase.sh proves the series sits on the upstream tip, is +# linear, lost or gained no commit, and changed the tree by exactly the +# upstream delta; +# - kernel-build.sh builds x86_64 and arm64 vmlinux with every downstream +# feature built in. +# +# If both pass and Claude's report asks for no decision +# (report-needs-decision.sh), the tree is force-pushed to the result, pinned +# by lease to the tip the rebase started from, with the old tip kept as a +# backup branch. Otherwise, or if that push is refused, the result goes up as +# a rebase//... branch with a pull request carrying Claude's report; +# approving that pull request lands it (rebase-land.yml). A night where +# Claude could not produce a branch at all opens an issue instead. +# +# Nights with nothing new upstream stop in the first job and never start the +# model. +# +# Every job clones with --filter=blob:none: the kernel's full history with +# blobs is several gigabytes, and the jobs need the commit graph but only the +# contents of the files the series and the upstream delta touch. + +on: + workflow_call: + inputs: + downstream: + description: The tree to rebase, e.g. edera/mainline + type: string + required: true + key: + description: A short name for the tree, unique per run, for artifact names + type: string + required: true + upstream_url: + type: string + required: true + upstream_branch: + type: string + required: true + force: + description: Rebase even if an open rebase pull request already covers this upstream tip + type: boolean + default: false + +permissions: + contents: read + +env: + DOWNSTREAM: ${{ inputs.downstream }} + UPSTREAM_URL: ${{ inputs.upstream_url }} + UPSTREAM_BRANCH: ${{ inputs.upstream_branch }} + +jobs: + prepare: + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + contents: read + pull-requests: read + outputs: + run: ${{ steps.plan.outputs.run }} + old: ${{ steps.plan.outputs.old }} + up: ${{ steps.plan.outputs.up }} + prefix: ${{ steps.plan.outputs.prefix }} + candidate: ${{ steps.plan.outputs.candidate }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ env.DOWNSTREAM }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Decide whether tonight has work + id: plan + env: + GH_TOKEN: ${{ github.token }} + FORCE: ${{ inputs.force }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + old=$(git rev-parse HEAD) + up=$(git rev-parse FETCH_HEAD) + prefix="rebase/${DOWNSTREAM//\//-}/" + candidate="${prefix}$(date -u +%Y%m%d)-${old:0:12}-${up:0:12}" + { + echo "old=$old" + echo "up=$up" + echo "prefix=$prefix" + echo "candidate=$candidate" + } >> "$GITHUB_OUTPUT" + + skip() { + echo "run=false" >> "$GITHUB_OUTPUT" + echo "### Upstream rebase of \`$DOWNSTREAM\`" >> "$GITHUB_STEP_SUMMARY" + echo "$1" >> "$GITHUB_STEP_SUMMARY" + } + + if git merge-base --is-ancestor "$up" "$old"; then + skip "Nothing to do: \`$DOWNSTREAM\` (\`${old:0:12}\`) already contains \`$UPSTREAM_BRANCH\` (\`${up:0:12}\`)." + exit 0 + fi + + # The candidate name records both tips, so an open pull request for + # the same pair means last night's result is still awaiting review. + if [ "$FORCE" != "true" ]; then + pending=$(gh pr list --repo "$GITHUB_REPOSITORY" --state open --base "$DOWNSTREAM" \ + --json headRefName,url \ + -q ".[] | select(.headRefName | startswith(\"$prefix\") and endswith(\"-${old:0:12}-${up:0:12}\")) | .url") + if [ -n "$pending" ]; then + skip "A rebase of these exact tips is already awaiting review: $pending" + exit 0 + fi + fi + echo "run=true" >> "$GITHUB_OUTPUT" + + rebase: + needs: prepare + if: needs.prepare.outputs.run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 330 + # Read-only on purpose: the model runs here. It reaches the Claude API + # through OIDC federation, and that is the only thing id-token is for. + permissions: + contents: read + id-token: write + outputs: + produced: ${{ steps.collect.outputs.produced }} + result: ${{ steps.collect.outputs.result }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + # The skill and scripts come from the commit this run started from, + # never from the tree under test. The kernel's .gitignore ignores + # dot-directories, so .rebase-tools/ cannot end up in a commit. + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir .rebase-tools + git archive "$GITHUB_SHA" .github/scripts .automation | tar -x -C .rebase-tools --strip-components=1 + + - name: Fetch upstream + env: + UP: ${{ needs.prepare.outputs.up }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git branch upstream-target "$UP" + # The rebase commits; this identity is what the committer field shows. + git config --global user.name "edera-upstream-rebase" + git config --global user.email "noreply@edera.dev" + + - name: Install toolchains + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends \ + build-essential bc bison flex libelf-dev libssl-dev python3 \ + gcc-aarch64-linux-gnu + + - name: Normalize identifiers + id: ids + env: + RULE: ${{ vars.REBASE_FEDERATION_RULE_ID }} + ORG: ${{ vars.ANTHROPIC_ORGANIZATION_ID }} + SVC: ${{ vars.ANTHROPIC_SERVICE_ACCOUNT_ID }} + WS: ${{ vars.ANTHROPIC_WORKSPACE_ID }} + run: | + set -euo pipefail + strip() { printf '%s' "$1" | tr -d '[:space:]'; } + { + echo "rule=$(strip "$RULE")" + echo "org=$(strip "$ORG")" + echo "svc=$(strip "$SVC")" + echo "ws=$(strip "$WS")" + } >> "$GITHUB_OUTPUT" + # An unset variable otherwise surfaces as an opaque API failure a + # second into the model step, with nothing in the log to say why. + missing=0 + for n in REBASE_FEDERATION_RULE_ID:RULE ANTHROPIC_ORGANIZATION_ID:ORG ANTHROPIC_SERVICE_ACCOUNT_ID:SVC ANTHROPIC_WORKSPACE_ID:WS; do + eval "v=\$${n#*:}" + if [ -z "$(strip "$v")" ]; then + echo "::error::vars.${n%%:*} is empty or not visible to this repository." + missing=1 + fi + done + exit "$missing" + + - name: Rebase + id: claude + continue-on-error: true + uses: anthropics/claude-code-action@6fed3ca145920b639991cb756090506e1bcaf515 # v1.0.245 + with: + # The job token, which is read-only here. Without it the action + # would exchange OIDC for its own app token, which can write. + github_token: ${{ github.token }} + anthropic_federation_rule_id: '${{ steps.ids.outputs.rule }}' + anthropic_organization_id: '${{ steps.ids.outputs.org }}' + anthropic_service_account_id: '${{ steps.ids.outputs.svc }}' + anthropic_workspace_id: '${{ steps.ids.outputs.ws }}' + prompt: | + Rebase the downstream kernel series onto upstream by following + .rebase-tools/skills/upstream-rebase/SKILL.md exactly. Read it in + full before you start. + + DOWNSTREAM=${{ env.DOWNSTREAM }} + OLD_TIP=${{ needs.prepare.outputs.old }} (${{ env.DOWNSTREAM }}, checked out) + UPSTREAM=${{ needs.prepare.outputs.up }} (${{ env.UPSTREAM_BRANCH }}, local branch upstream-target) + RESULT=rebase-result + REPORT=${{ runner.temp }}/rebase-report.md + + You cannot push and must not try. The workflow verifies and + builds your branch independently afterwards and decides what + happens to it. + claude_args: | + --model claude-opus-5-5 + --max-turns 400 + --allowedTools "Read,Grep,Glob,Edit,Write,Bash(git:*),Bash(make:*),Bash(bash .rebase-tools/scripts/verify-upstream-rebase.sh:*),Bash(bash .rebase-tools/scripts/kernel-build.sh:*),Bash(scripts/config:*),Bash(mkdir -p .rebase-scratch:*),Bash(diff:*),Bash(comm:*),Bash(sort:*),Bash(grep:*),Bash(sed:*),Bash(awk:*),Bash(cat:*),Bash(head:*),Bash(tail:*),Bash(wc:*),Bash(ls:*),Bash(cut:*),Bash(tee:*)" + + # Take only what Claude left in git and the report file. A branch that + # does not exist or does not sit on the upstream tip is a failed night. + - name: Collect the result + id: collect + if: always() + env: + REPORT: ${{ runner.temp }}/rebase-report.md + UP: ${{ needs.prepare.outputs.up }} + OUTCOME: ${{ steps.claude.outcome }} + EXECUTION_FILE: ${{ steps.claude.outputs.execution_file }} + run: | + set -euo pipefail + out=$RUNNER_TEMP/out + mkdir -p "$out" + if [ -s "$REPORT" ]; then + cp "$REPORT" "$out/report.md" + else + # The action hides the session to keep tool output out of a + # public log, which also hides why a run ended early. Take only + # the API error messages and the final result text from it: + # neither carries tool output. + why= + f=${EXECUTION_FILE:-$RUNNER_TEMP/claude-execution-output.json} + if [ -s "$f" ]; then + why=$(jq -r ' + (if type == "array" then .[] else . end) + | if .type == "assistant" and (.error != null) then + "API error (\(.error)): " + ([.message.content[]? | .text? // empty] | join(" ")) + elif .type == "result" then + ([.result // empty] + (.errors // [] | map(tostring))) | join("\n") + else empty end' "$f" 2>/dev/null | head -c 2000 || true) + fi + { + printf 'Claude left no report (rebase step: %s).\n' "$OUTCOME" + if [ -n "$why" ]; then + # shellcheck disable=SC2016 # a literal Markdown fence + printf '\n```\n%s\n```\n' "$why" + else + printf 'The run log has the cause.\n' + fi + } >"$out/report.md" + [ -z "$why" ] || echo "::error::Claude ended without a result: ${why//$'\n'/ }" + fi + if git rev-parse --verify -q refs/heads/rebase-result >/dev/null \ + && git merge-base --is-ancestor "$UP" rebase-result \ + && [ "$(git rev-parse rebase-result)" != "$UP" ]; then + git bundle create "$out/rebase.bundle" refs/heads/rebase-result "^$UP" + echo "produced=true" >> "$GITHUB_OUTPUT" + echo "result=$(git rev-parse rebase-result)" >> "$GITHUB_OUTPUT" + else + echo "produced=false" >> "$GITHUB_OUTPUT" + fi + + - name: Upload + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/out/ + if-no-files-found: error + retention-days: 30 + + verify: + needs: [prepare, rebase] + if: needs.rebase.outputs.produced == 'true' + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + status: ${{ steps.check.outputs.status }} + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + # The checker comes from the commit this run started from, never from + # the branch under test. + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in + + - name: Check + id: check + env: + OLD: ${{ needs.prepare.outputs.old }} + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$RUNNER_TEMP/in/rebase.bundle" rebase-result:rebase-result + [ "$(git rev-parse rebase-result)" = "$RESULT" ] || { echo "bundle does not hold $RESULT"; exit 1; } + # Actions runs this step under bash -e, so a bare call would end the + # step on the checker's first non-zero exit, before its verdict is + # recorded: drift would then reach the pull request as "the checker + # did not run". Capture the status instead. + rc=0 + bash "$RUNNER_TEMP/tools/scripts/verify-upstream-rebase.sh" "$OLD" "$RESULT" "$UP" >"$RUNNER_TEMP/verify.md" || rc=$? + case $rc in + 0) status=clean ;; + 1) status=drift ;; + *) status=broken ;; + esac + echo "status=$status" >> "$GITHUB_OUTPUT" + cat "$RUNNER_TEMP/verify.md" >> "$GITHUB_STEP_SUMMARY" + + - name: Upload + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: verify-${{ inputs.key }} + path: ${{ runner.temp }}/verify.md + retention-days: 30 + + build: + needs: [prepare, rebase] + if: needs.rebase.outputs.produced == 'true' + runs-on: ubuntu-latest + timeout-minutes: 120 + strategy: + fail-fast: false + matrix: + arch: [x86_64, arm64] + name: build (${{ matrix.arch }}) + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in + + - name: Check out the result + env: + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + run: | + set -euo pipefail + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$RUNNER_TEMP/in/rebase.bundle" rebase-result:rebase-result + git checkout --detach "$RESULT" + + - name: Install toolchains + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends \ + build-essential bc bison flex libelf-dev libssl-dev python3 \ + gcc-aarch64-linux-gnu + + - name: Build + env: + ARCH: ${{ matrix.arch }} + run: bash "$RUNNER_TEMP/tools/scripts/kernel-build.sh" "$ARCH" "$RUNNER_TEMP/obj" + + publish: + needs: [prepare, rebase, verify, build] + if: always() && needs.prepare.outputs.run == 'true' && needs.rebase.result != 'cancelled' + runs-on: ubuntu-latest + timeout-minutes: 45 + concurrency: + group: push-${{ inputs.downstream }} + cancel-in-progress: false + permissions: + contents: read + issues: write + steps: + - name: Harden runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ needs.prepare.outputs.old }} + fetch-depth: 0 + filter: blob:none + persist-credentials: false + + - name: Trusted tools + run: | + set -euo pipefail + git fetch --no-tags --filter=blob:none origin "$GITHUB_SHA" + mkdir "$RUNNER_TEMP/tools" + git archive "$GITHUB_SHA" .github/scripts | tar -x -C "$RUNNER_TEMP/tools" --strip-components=1 + + - name: Download the rebase + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rebase-${{ inputs.key }} + path: ${{ runner.temp }}/in/rebase + + # Absent when Claude produced no branch and the checker never ran. + - name: Download the checker report + if: needs.verify.result == 'success' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: verify-${{ inputs.key }} + path: ${{ runner.temp }}/in/verify + + - name: App token + id: app + if: needs.rebase.outputs.produced == 'true' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.REBASE_APP_CLIENT_ID }} + private-key: ${{ secrets.REBASE_APP_PRIVATE_KEY }} + permission-contents: write + permission-pull-requests: write + permission-workflows: write + + - name: Push or propose + env: + APP_TOKEN: ${{ steps.app.outputs.token }} + JOB_TOKEN: ${{ github.token }} + PRODUCED: ${{ needs.rebase.outputs.produced }} + VERIFY: ${{ needs.verify.outputs.status }} + BUILD: ${{ needs.build.result }} + OLD: ${{ needs.prepare.outputs.old }} + UP: ${{ needs.prepare.outputs.up }} + RESULT: ${{ needs.rebase.outputs.result }} + PREFIX: ${{ needs.prepare.outputs.prefix }} + CANDIDATE: ${{ needs.prepare.outputs.candidate }} + KEY: ${{ inputs.key }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + in=$RUNNER_TEMP/in + report=$in/rebase/report.md + verify=$in/verify/verify.md + summary() { cat >> "$GITHUB_STEP_SUMMARY"; } + title="Nightly upstream rebase of $DOWNSTREAM failed" + + # One issue per tree that stays open until someone deals with it, + # rather than one per night. + report_issue() { + existing=$(GH_TOKEN=$JOB_TOKEN gh issue list --repo "$GITHUB_REPOSITORY" --state open \ + --search "in:title \"$title\"" --json number -q '.[0].number // empty') + if [ -n "$existing" ]; then + GH_TOKEN=$JOB_TOKEN gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body-file "$1" + else + GH_TOKEN=$JOB_TOKEN gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$1" + fi + } + + if [ "$PRODUCED" != "true" ]; then + body=$RUNNER_TEMP/issue.md + { + echo "Claude could not produce a rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`). [Run log]($RUN_URL)." + echo + cat "$report" + } | head -c 60000 >"$body" + report_issue "$body" + echo "No branch produced; reported in an issue." | summary + exit 1 + fi + + git fetch --no-tags "$UPSTREAM_URL" "$UPSTREAM_BRANCH" + git merge-base --is-ancestor "$UP" FETCH_HEAD + git fetch "$in/rebase/rebase.bundle" rebase-result:rebase-result + [ "$(git rev-parse rebase-result)" = "$RESULT" ] + + # Workflows triggered by a pull request, including the land workflow, + # run that branch's own copy of .github/ with this repository's + # secrets. So the automation paths must change by exactly what + # upstream changed there, or the branch never leaves this job: not + # as the tree and not as a pull request. + mb=$(git merge-base "$OLD" "$UP") + guarded=(.github .automation) + lines() { git diff "$@" -- "${guarded[@]}" | grep -E '^([+-]|diff )' || true; } + if [ "$(lines "$OLD" "$RESULT")" != "$(lines "$mb" "$UP")" ]; then + body=$RUNNER_TEMP/issue.md + { + echo "**Refused to publish** the rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`): it changes \`${guarded[*]}\` beyond what upstream changed there. Those paths run with repository secrets, so the branch was not pushed anywhere. It is in the run's \`rebase-${KEY}\` artifact. [Run log]($RUN_URL)." + echo + echo '```diff' + git diff --stat "$OLD" "$RESULT" -- "${guarded[@]}" + echo '```' + } >"$body" + report_issue "$body" + echo "::error::Rebase changes automation paths beyond the upstream delta; not published." + exit 1 + fi + + origin="https://x-access-token:${APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + export GH_TOKEN=$APP_TOKEN + + close_superseded() { + gh pr list --repo "$GITHUB_REPOSITORY" --state open --base "$DOWNSTREAM" \ + --json number,headRefName \ + -q ".[] | select(.headRefName | startswith(\"$PREFIX\")) | select(.headRefName != \"$CANDIDATE\") | .number" | + while read -r n; do + gh pr close "$n" --repo "$GITHUB_REPOSITORY" --delete-branch \ + --comment "Superseded by $1." + done + } + + # Reasons a human has to look before the tree moves. The checker + # and the builds judge the branch; the report can still raise a + # question they cannot see, such as an upstream change that meets a + # downstream feature without touching the same lines. + why=() + [ "$VERIFY" = "clean" ] || why+=("the checker found ${VERIFY:-no result (it did not run)}") + [ "$BUILD" = "success" ] || why+=("a build ${BUILD}") + drc=0 + bash "$RUNNER_TEMP/tools/scripts/report-needs-decision.sh" "$report" >/dev/null || drc=$? + case $drc in + 0) why+=("Claude's report asks for a decision") ;; + 1) ;; + *) why+=("Claude's report could not be read") ;; + esac + + if [ "${#why[@]}" -eq 0 ]; then + backup="backup/${DOWNSTREAM//\//-}-pre-rebase-$(date -u +%Y%m%d)-${OLD:0:12}" + git push "$origin" "$OLD:refs/heads/$backup" + # The lease pins the tip this rebase started from: if anything + # landed on the tree since, the push fails and nothing is lost. + if git push --force-with-lease="refs/heads/$DOWNSTREAM:$OLD" "$origin" \ + "$RESULT:refs/heads/$DOWNSTREAM" 2>"$RUNNER_TEMP/push.err"; then + cat "$RUNNER_TEMP/push.err" >&2 + now=$(git ls-remote "$origin" "refs/heads/$DOWNSTREAM" | cut -f1) + [ "$now" = "$RESULT" ] || { echo "::error::$DOWNSTREAM is $now after the push, expected $RESULT"; exit 1; } + close_superseded "the clean rebase pushed to \`$DOWNSTREAM\` as \`${RESULT:0:12}\`" + { + echo "### Pushed \`$DOWNSTREAM\` → \`${RESULT:0:12}\`" + echo "Was \`${OLD:0:12}\`, kept as \`$backup\`." + echo + cat "$verify" + echo + cat "$report" + } | summary + exit 0 + fi + # Refused: a branch rule, or the tree moved under the lease. A + # clean result is still worth a reviewer's approval, so propose + # it rather than drop it; the land workflow re-checks the tip. + cat "$RUNNER_TEMP/push.err" >&2 + reason=$(grep -m1 -E 'GH0[0-9]+|\[(remote )?rejected\]|error:' "$RUNNER_TEMP/push.err" | + sed -E 's/^remote: *//; s/^[[:space:]]+//; s/[[:space:]]+$//' || true) + why+=("the push to \`$DOWNSTREAM\` was refused${reason:+ (\`$reason\`)}") + git push "$origin" --delete "$backup" || true + fi + + # Not pushed: propose it. + git push "$origin" "$RESULT:refs/heads/$CANDIDATE" + because=$(IFS=';'; echo "${why[*]}") + because=${because//;/ and } + body=$RUNNER_TEMP/pr.md + { + echo "Nightly rebase of \`$DOWNSTREAM\` (\`${OLD:0:12}\`) onto \`$UPSTREAM_BRANCH\` (\`${UP:0:12}\`), driven by Claude. **Not pushed automatically because ${because}.** [Run log]($RUN_URL)." + echo + echo "> [!IMPORTANT]" + echo "> Do not use the merge button: this branch replaces \`$DOWNSTREAM\` rather than merging into it. **Approve** the pull request to land it; the land workflow force-pushes \`$DOWNSTREAM\` to the approved head, pinned to \`${OLD:0:12}\`, and keeps a backup." + echo + echo "## Independent checks" + echo + echo "Builds: **${BUILD}** (x86_64 and arm64 vmlinux with the Edera features built in; per-build results in the run)." + echo + if [ -s "$verify" ]; then cat "$verify"; else echo "The checker did not run."; fi + echo + echo "# Claude's report" + echo + cat "$report" + } >"$body" + if [ "$(wc -c <"$body")" -gt 60000 ]; then + head -c 59000 "$body" >"$body.cut" + printf '\n\n_Truncated; the full report is in the run artifacts._\n' >>"$body.cut" + mv "$body.cut" "$body" + fi + url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$DOWNSTREAM" --head "$CANDIDATE" \ + --title "Rebase $DOWNSTREAM onto $UPSTREAM_BRANCH @ ${UP:0:12}" --body-file "$body") + close_superseded "$url" + echo "### Proposed: $url" | summary diff --git a/.gitignore b/.gitignore index 9875120ea7bde0..f0351b8a74d0e0 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,7 @@ *.gcno *.gcda *.gz +*.header-check *.i *.ko *.lex.c @@ -75,7 +76,6 @@ modules.order /vmlinux-gdb.py /vmlinuz /System.map -/Module.markers /modules.builtin /modules.builtin.modinfo /modules.builtin.ranges @@ -104,6 +104,7 @@ modules.order # # pacman files (make pacman-pkg) # +/*.pkg.*.sig /PKGBUILD /pacman/ diff --git a/.mailmap b/.mailmap index 0a0be49bc4de6c..abb09aab44c6fd 100644 --- a/.mailmap +++ b/.mailmap @@ -321,6 +321,7 @@ Gao Xiang Gao Xiang Gao Xiang Gao Xiang +Gao Xiang Geliang Tang Geliang Tang Geliang Tang diff --git a/AGENTS.md b/AGENTS.md new file mode 120000 index 00000000000000..100b93820ade4c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1 @@ +README \ No newline at end of file diff --git a/CREDITS b/CREDITS index a0bd941ca5dd26..8bdfde15e2ace8 100644 --- a/CREDITS +++ b/CREDITS @@ -57,7 +57,7 @@ S: Longford, Ireland S: Sydney, Australia N: Tigran A. Aivazian -E: tigran@aivazian.fsnet.co.uk +E: aivazian.tigran@gmail.com W: http://www.moses.uklinux.net/patches D: BFS filesystem D: Intel IA32 CPU microcode update support diff --git a/Documentation/ABI/stable/sysfs-driver-firmware-zynqmp b/Documentation/ABI/stable/sysfs-driver-firmware-zynqmp index c3fec3c835af36..214e9ed4564e55 100644 --- a/Documentation/ABI/stable/sysfs-driver-firmware-zynqmp +++ b/Documentation/ABI/stable/sysfs-driver-firmware-zynqmp @@ -141,32 +141,44 @@ Description: Usage: - Select over temperature config ID to enable/disable feature + Select over temperature config ID to enable/disable feature:: + # echo 1 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id - Check over temperature config ID is selected or not + Check over temperature config ID is selected or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id + The expected result is 1. - Select over temperature config ID to configure OT limit + Select over temperature config ID to configure OT limit:: + # echo 2 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id - Check over temperature config ID is selected or not + Check over temperature config ID is selected or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id + The expected result is 2. - Select external watchdog config ID to enable/disable feature + Select external watchdog config ID to enable/disable feature:: + # echo 3 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id - Check external watchdog config ID is selected or not + Check external watchdog config ID is selected or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id + The expected result is 3. - Select external watchdog config ID to configure time interval + Select external watchdog config ID to configure time interval:: + # echo 4 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id - Check external watchdog config ID is selected or not + Check external watchdog config ID is selected or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id + The expected result is 4. Users: Xilinx @@ -205,52 +217,70 @@ Description: Usage: - Enable over temperature feature + Enable over temperature feature:: + # echo 1 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 1 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the over temperature feature is enabled or not + Check whether the over temperature feature is enabled or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 1. - Disable over temperature feature + Disable over temperature feature:: + # echo 1 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 0 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the over temperature feature is disabled or not + Check whether the over temperature feature is disabled or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 0. - Configure over temperature limit to 50 Degree Celsius + Configure over temperature limit to 50 Degree Celsius:: + # echo 2 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 50 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the over temperature limit is configured or not + Check whether the over temperature limit is configured or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 50. - Enable external watchdog feature + Enable external watchdog feature:: + # echo 3 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 1 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the external watchdog feature is enabled or not + Check whether the external watchdog feature is enabled or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 1. - Disable external watchdog feature + Disable external watchdog feature:: + # echo 3 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 0 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the external watchdog feature is disabled or not + Check whether the external watchdog feature is disabled or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 0. - Configure external watchdog timer interval to 500ms + Configure external watchdog timer interval to 500ms:: + # echo 4 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_id # echo 500 > /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value - Check whether the external watchdog timer interval is configured or not + Check whether the external watchdog timer interval is configured or not:: + # cat /sys/devices/platform/firmware\:zynqmp-firmware/feature_config_value + The expected result is 500. Users: Xilinx diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index 55ea1db749a180..14a8667156123d 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -486,19 +486,19 @@ Description: These files provide an interface to PCIe Resizable BAR support. A file is created for each BAR resource (N) supported by the PCIe Resizable BAR extended capability of the device. Reading - each file exposes the bitmap of available resource sizes: + each file exposes the bitmap of available resource sizes:: - # cat resource1_resize - 00000000000001c0 + # cat resource1_resize + 00000000000001c0 The bitmap represents supported resource sizes for the BAR, where bit0 = 1MB, bit1 = 2MB, bit2 = 4MB, etc. In the above example the device supports 64MB, 128MB, and 256MB BAR sizes. When writing the file, the user provides the bit position of - the desired resource size, for example: + the desired resource size, for example:: - # echo 7 > resource1_resize + # echo 7 > resource1_resize This indicates to set the size value corresponding to bit 7, 128MB. The resulting size is 2 ^ (bit# + 20). This definition @@ -600,7 +600,7 @@ Description: As all DOE devices must support the DOE discovery feature, if DOE is supported you will at least see the doe_discovery - file, with this contents: + file, with this contents:: # cat doe_features/doe_discovery 0001:00 @@ -608,7 +608,7 @@ Description: If the device supports other features you will see other files as well. For example if CMA/SPDM and secure CMA/SPDM are supported the doe_features directory will look like - this: + this:: # ls doe_features 0001:01 0001:02 doe_discovery diff --git a/Documentation/ABI/testing/sysfs-bus-usb b/Documentation/ABI/testing/sysfs-bus-usb index baf4bf6d32ee37..5213117fd957e6 100644 --- a/Documentation/ABI/testing/sysfs-bus-usb +++ b/Documentation/ABI/testing/sysfs-bus-usb @@ -83,7 +83,8 @@ Description: idVendor idProduct. After successfully removing an ID, the driver will no longer support the device. This is useful to ensure auto probing won't - match the driver to the device. For example: + match the driver to the device. For example:: + # echo "046d c315" > /sys/bus/usb/drivers/foo/remove_id Reading from this file will list the dynamically added diff --git a/Documentation/ABI/testing/sysfs-class-firmware-attributes b/Documentation/ABI/testing/sysfs-class-firmware-attributes index 2713efa509b465..40e35fe97a7d82 100644 --- a/Documentation/ABI/testing/sysfs-class-firmware-attributes +++ b/Documentation/ABI/testing/sysfs-class-firmware-attributes @@ -397,22 +397,25 @@ Description: unlimited attributes. Read the attribute to check what save mode is enabled (single or bulk). - E.g: - # cat /sys/class/firmware-attributes/thinklmi/attributes/save_settings - single + E.g:: + + # cat /sys/class/firmware-attributes/thinklmi/attributes/save_settings + single Write the attribute with 'bulk' to enable bulk save mode. - Write the attribute with 'single' to enable saving, after every attribute set. + Write the attribute with 'single' to enable saving after every attribute set. The default setting is single mode. - E.g: - # echo bulk > /sys/class/firmware-attributes/thinklmi/attributes/save_settings + E.g:: + + # echo bulk > /sys/class/firmware-attributes/thinklmi/attributes/save_settings When in bulk mode write 'save' to trigger a save of all currently modified attributes. - Note, once a save has been triggered, in bulk mode, attributes can no longer be set and + Note, once a save has been triggered in bulk mode, attributes can no longer be set and will return a permissions error. This is to prevent users hitting the 48+ save limitation (which requires entering the BIOS to clear the error condition) - E.g: - # echo save > /sys/class/firmware-attributes/thinklmi/attributes/save_settings + E.g:: + + # echo save > /sys/class/firmware-attributes/thinklmi/attributes/save_settings What: /sys/class/firmware-attributes/*/attributes/debug_cmd Date: July 2021 diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu index 82d10d556cc89f..545687ebadccc7 100644 --- a/Documentation/ABI/testing/sysfs-devices-system-cpu +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu @@ -335,12 +335,16 @@ Description: Performance Limited Read to check if platform throttling (thermal/power/current limits) caused delivered performance to fall below the requested level. A non-zero value indicates throttling occurred. + If firmware provides the register but Linux cannot access it + safely, reads return the literal string "". - Write the bitmask of bits to clear: - - - 0x1 = clear bit 0 (desired performance excursion) - - 0x2 = clear bit 1 (minimum performance excursion) - - 0x3 = clear both bits + Write 0x3 to clear both bits. Selectively clearing one bit is + not supported because ACPI does not define the effect of writing + one to these status bits and an interlocked read-modify-write is + not generally available. + Writing 0x3 returns -EOPNOTSUPP when firmware describes the + register using an access which Linux cannot clear safely; reads + may remain available in that case. The platform sets these bits; OSPM can only clear them. @@ -689,15 +693,21 @@ Description: Umwait control Low order two bits must be zero. What: /sys/devices/system/cpu/sev + /sys/devices/system/cpu/sev/sev_status /sys/devices/system/cpu/sev/vmpl Date: May 2024 Contact: Linux kernel mailing list Description: Secure Encrypted Virtualization (SEV) information - This directory is only present when running as an SEV-SNP guest. + This directory is only present when running as an SEV guest. + + sev_status: Reports the value of the SEV_STATUS MSR which + enumerates the enabled features of an SEV + environment. vmpl: Reports the Virtual Machine Privilege Level (VMPL) at which - the SEV-SNP guest is running. + the SEV-SNP guest is running. This file is only present + when running as an SEV-SNP guest. What: /sys/devices/system/cpu/svm @@ -806,3 +816,17 @@ Date: Nov 2022 Contact: Linux kernel mailing list Description: (RO) the list of CPUs that can be brought online. + +What: /sys/devices/system/cpu/preferred +Date: Sep 2026 +Contact: Linux kernel mailing list +Description: + (RO) the list of preferred CPUs applicable in + paravirtualized environments. + + The steal governor driver dynamically adjusts this mask + based on observed steal time. Scheduling tasks on + CPUs outside of this list may lead to performance + degradations due to underlying physical CPU contention. + + See Documentation/scheduler/sched-paravirt.rst for more details. diff --git a/Documentation/ABI/testing/sysfs-driver-aspeed-uart-routing b/Documentation/ABI/testing/sysfs-driver-aspeed-uart-routing index 910df0e5815abf..b0b713c855175f 100644 --- a/Documentation/ABI/testing/sysfs-driver-aspeed-uart-routing +++ b/Documentation/ABI/testing/sysfs-driver-aspeed-uart-routing @@ -8,9 +8,10 @@ Description: Selects the RX source of the UARTx device. selected option marked by brackets "[]". The list of available options depends on the selected file. - e.g. - cat /sys/bus/platform/drivers/aspeed-uart-routing/\*.uart_routing/uart1 - [io1] io2 io3 io4 uart2 uart3 uart4 io6 + E.g.:: + + cat /sys/bus/platform/drivers/aspeed-uart-routing/\*.uart_routing/uart1 + [io1] io2 io3 io4 uart2 uart3 uart4 io6 In this case, UART1 gets its input from IO1 (physical serial port 1). diff --git a/Documentation/ABI/testing/sysfs-driver-xdata b/Documentation/ABI/testing/sysfs-driver-xdata index f574e8e6dca21d..5bcdcba3a5c95f 100644 --- a/Documentation/ABI/testing/sysfs-driver-xdata +++ b/Documentation/ABI/testing/sysfs-driver-xdata @@ -9,15 +9,19 @@ Description: Allows the user to enable the PCIe traffic generator which Write y/1/on to enable, n/0/off to disable - Usage e.g. + Usage e.g.:: + echo 1 > /sys/class/misc/dw-xdata-pcie./write - or + + or:: + echo 0 > /sys/class/misc/dw-xdata-pcie./write The user can read the current PCIe link throughput generated through this generator in MB/s. - Usage e.g. + Usage e.g.:: + cat /sys/class/misc/dw-xdata-pcie./write 204 @@ -34,15 +38,19 @@ Description: Allows the user to enable the PCIe traffic generator which Write y/1/on to enable, n/0/off to disable - Usage e.g. + Usage e.g.:: + echo 1 > /sys/class/misc/dw-xdata-pcie./read - or + + or:: + echo 0 > /sys/class/misc/dw-xdata-pcie./read The user can read the current PCIe link throughput generated through this generator in MB/s. - Usage e.g. + Usage e.g.:: + cat /sys/class/misc/dw-xdata-pcie./read 199 diff --git a/Documentation/ABI/testing/sysfs-driver-xilinx-tmr-manager b/Documentation/ABI/testing/sysfs-driver-xilinx-tmr-manager index 57b9b68a73ee9a..8d07c982d125f9 100644 --- a/Documentation/ABI/testing/sysfs-driver-xilinx-tmr-manager +++ b/Documentation/ABI/testing/sysfs-driver-xilinx-tmr-manager @@ -3,14 +3,16 @@ Date: Nov 2022 Contact: appana.durga.kedareswara.rao@amd.com Description: This control file provides the fault detection count. This file cannot be written. - Example: - # cat /sys/devices/platform/amba_pl/44a10000.tmr_manager/errcnt - 1 + Example:: + + # cat /sys/devices/platform/amba_pl/44a10000.tmr_manager/errcnt + 1 What: /sys/devices/platform/amba_pl//dis_block_break Date: Nov 2022 Contact: appana.durga.kedareswara.rao@amd.com -Description: Write any value to it, This control file enables the break signal. +Description: Write any value to it. This control file enables the break signal. This file is write only. - Example: - # echo > /sys/devices/platform/amba_pl/44a10000.tmr_manager/dis_block_break + Example:: + + # echo > /sys/devices/platform/amba_pl/44a10000.tmr_manager/dis_block_break diff --git a/Documentation/ABI/testing/sysfs-firmware-cca b/Documentation/ABI/testing/sysfs-firmware-cca new file mode 100644 index 00000000000000..d545b752353049 --- /dev/null +++ b/Documentation/ABI/testing/sysfs-firmware-cca @@ -0,0 +1,7 @@ +What: /sys/firmware/cca/realm_guest +Date: September 2026 +Contact: Linux ARM Kernel Mailing list +Description: Read-only. Present only when the kernel is running as an + Arm Confidential Compute Architecture (CCA) Realm guest, in + which case it reads 1. If the file does not exist, the kernel + is not running as a Realm guest. diff --git a/Documentation/ABI/testing/sysfs-platform-intel-ifs b/Documentation/ABI/testing/sysfs-platform-intel-ifs index 41b4d5b1e21c6d..11cd2f412d8172 100644 --- a/Documentation/ABI/testing/sysfs-platform-intel-ifs +++ b/Documentation/ABI/testing/sysfs-platform-intel-ifs @@ -10,8 +10,10 @@ Description: Write to trigger IFS test for one online core. Note that the test is per core. The cpu# can be for any thread on the core. Running on one thread completes the test for the core containing that thread. - Example: to test the core containing cpu5: echo 5 > - /sys/devices/virtual/misc/intel_ifs_/run_test + Example: to test the core containing cpu5:: + + echo 5 > /sys/devices/virtual/misc/intel_ifs_/run_test + Devices: all What: /sys/devices/virtual/misc/intel_ifs_/status diff --git a/Documentation/ABI/testing/sysfs-platform-ts5500 b/Documentation/ABI/testing/sysfs-platform-ts5500 deleted file mode 100644 index e685957caa1236..00000000000000 --- a/Documentation/ABI/testing/sysfs-platform-ts5500 +++ /dev/null @@ -1,54 +0,0 @@ -What: /sys/devices/platform/ts5500/adc -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Indicates the presence of an A/D Converter. If it is present, - it will display "1", otherwise "0". - -What: /sys/devices/platform/ts5500/ereset -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Indicates the presence of an external reset. If it is present, - it will display "1", otherwise "0". - -What: /sys/devices/platform/ts5500/id -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Product ID of the TS board. TS-5500 ID is 0x60. - -What: /sys/devices/platform/ts5500/jumpers -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Bitfield showing the jumpers' state. If a jumper is present, - the corresponding bit is set. For instance, 0x0e means jumpers - 2, 3 and 4 are set. - -What: /sys/devices/platform/ts5500/name -Date: July 2014 -KernelVersion: 3.16 -Contact: "Savoir-faire Linux Inc." -Description: - Model name of the TS board, e.g. "TS-5500". - -What: /sys/devices/platform/ts5500/rs485 -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Indicates the presence of the RS485 option. If it is present, - it will display "1", otherwise "0". - -What: /sys/devices/platform/ts5500/sram -Date: January 2013 -KernelVersion: 3.7 -Contact: "Savoir-faire Linux Inc." -Description: - Indicates the presence of the SRAM option. If it is present, - it will display "1", otherwise "0". diff --git a/Documentation/Makefile b/Documentation/Makefile index 377a449656c817..ee852afe9bb998 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -74,6 +74,10 @@ htmldocs-redirects: $(srctree)/Documentation/.renames.txt refcheckdocs: $(Q)cd $(srctree); tools/docs/documentation-file-ref-check +testdocs: + $(Q)PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \ + $(PYTHON3) $(srctree)/tools/unittests/run.py + cleandocs: $(Q)rm -rf $(BUILDDIR) @@ -95,6 +99,7 @@ dochelp: @echo ' (will connect to external hosts)' @echo ' refcheckdocs - check for references to non-existing files under' @echo ' Documentation' + @echo ' testdocs - run the unit tests of the documentation tools' @echo ' cleandocs - clean all generated files' @echo @echo ' make SPHINXDIRS="s1 s2" [target] Generate only docs of folder s1, s2' diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.rst index d7c8eff6331740..a986358a5c2795 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -122,10 +122,11 @@ warnings: of RCU CPU stall warnings, eventually leading to the realization that the CPU had failed. -The RCU, RCU-sched, RCU-tasks, and RCU-tasks-trace implementations have -CPU stall warning. Note that SRCU does *not* have CPU stall warnings. -Please note that RCU only detects CPU stalls when there is a grace period -in progress. No grace period, no CPU stall warnings. +The RCU and RCU-tasks implementations have CPU stall warning. Note that +SRCU does *not* have CPU stall warnings, and now that RCU-tasks-trace +is implemented in terms of SRCU, neither does it. Please note that +RCU only detects CPU stalls when there is a grace period in progress. +No grace period, no CPU stall warnings. To diagnose the cause of the stall, inspect the stack traces. The offending function will usually be near the top of the stack. @@ -213,21 +214,16 @@ RCU_STALL_RAT_DELAY rcupdate.rcu_task_stall_timeout ------------------------------- - This boot/sysfs parameter controls the RCU-tasks and - RCU-tasks-trace stall warning intervals. A value of zero or less - suppresses RCU-tasks stall warnings. A positive value sets the - stall-warning interval in seconds. An RCU-tasks stall warning - starts with the line: + This boot/sysfs parameter controls the RCU-tasks stall warning + interval. A value of zero or less suppresses RCU-tasks stall + warnings. A positive value sets the stall-warning interval + in seconds. An RCU-tasks stall warning starts with the line: INFO: rcu_tasks detected stalls on tasks: And continues with the output of sched_show_task() for each task stalling the current RCU-tasks grace period. - An RCU-tasks-trace stall warning starts (and continues) similarly: - - INFO: rcu_tasks_trace detected stalls on tasks - Interpreting RCU's CPU Stall-Detector "Splats" ============================================== diff --git a/Documentation/admin-guide/LSM/LoadPin.rst b/Documentation/admin-guide/LSM/LoadPin.rst index dd3ca68b5df188..a90fd9dac9606f 100644 --- a/Documentation/admin-guide/LSM/LoadPin.rst +++ b/Documentation/admin-guide/LSM/LoadPin.rst @@ -23,9 +23,9 @@ sysctl allows for easy testing on systems with a mutable filesystem.) It's also possible to exclude specific file types from LoadPin using kernel command line option "``loadpin.exclude``". By default, all files are included, but they can be excluded using kernel command line option such -as "``loadpin.exclude=kernel-module,kexec-image``". This allows to use +as "``loadpin.exclude=kernel-module,kexec-image``". This allows using different mechanisms such as ``CONFIG_MODULE_SIG`` and -``CONFIG_KEXEC_VERIFY_SIG`` to verify kernel module and kernel image while -still use LoadPin to protect the integrity of other files kernel loads. The -full list of valid file types can be found in ``kernel_read_file_str`` -defined in ``include/linux/kernel_read_file.h``. +``CONFIG_KEXEC_SIG`` to verify kernel modules and kernel images while +still using LoadPin to protect the integrity of other files the kernel +loads. The full list of valid file types can be found in +``kernel_read_file_str`` defined in ``include/linux/kernel_read_file.h``. diff --git a/Documentation/admin-guide/LSM/index.rst b/Documentation/admin-guide/LSM/index.rst index b44ef68f6e4da6..9518495edfbc32 100644 --- a/Documentation/admin-guide/LSM/index.rst +++ b/Documentation/admin-guide/LSM/index.rst @@ -6,9 +6,11 @@ The Linux Security Module (LSM) framework provides a mechanism for various security checks to be hooked by new kernel extensions. The name "module" is a bit of a misnomer since these extensions are not actually loadable kernel modules. Instead, they are selectable at build-time via -CONFIG_DEFAULT_SECURITY and can be overridden at boot-time via the -``"security=..."`` kernel command line argument, in the case where multiple -LSMs were built into a given kernel. +CONFIG_LSM, an ordered list of the LSMs to enable, and can be +overridden at boot-time via the ``"lsm=..."`` kernel command line +argument. The ``"security=..."`` kernel command line argument remains +available to choose a legacy "major" security module, but has been +deprecated by the ``"lsm=..."`` parameter. The primary users of the LSM interface are Mandatory Access Control (MAC) extensions which provide a comprehensive security policy. Examples @@ -25,9 +27,15 @@ man-pages project. A list of the active security modules can be found by reading ``/sys/kernel/security/lsm``. This is a comma separated list, and will always include the capability module. The list reflects the -order in which checks are made. The capability module will always -be first, followed by any "minor" modules (e.g. Yama) and then -the one "major" module (e.g. SELinux) if there is one configured. +order in which checks are made. The capability module will be +first, unless CONFIG_SECURITY_LOCKDOWN_LSM_EARLY is enabled, in +which case the lockdown module will precede it. The integrity +modules (e.g. IMA and EVM), if enabled in the kernel +configuration, are always placed at the end of the list. Any +other "minor" modules (e.g. Yama) and the one "major" module +(e.g. SELinux), if there is one configured, appear in between, +in the order given by CONFIG_LSM or the ``"lsm=..."`` kernel +command line parameter. Process attributes associated with "major" security modules should be accessed and maintained using the special files in ``/proc/.../attr``. diff --git a/Documentation/admin-guide/RAS/main.rst b/Documentation/admin-guide/RAS/main.rst index 5a45db32c49b09..c1040e5a5b3537 100644 --- a/Documentation/admin-guide/RAS/main.rst +++ b/Documentation/admin-guide/RAS/main.rst @@ -623,7 +623,7 @@ Under ``/sys/devices/system/edac/pci`` are control and attribute files as follows: -- ``check_pci_parity`` - Enable/Disable PCI Parity checking control file +- ``check_pci_errors`` - Enable/Disable PCI Parity checking control file This control file enables or disables the PCI Bus Parity scanning operation. Writing a 1 to this file enables the scanning. Writing @@ -631,11 +631,11 @@ follows: Enable:: - echo "1" >/sys/devices/system/edac/pci/check_pci_parity + echo "1" >/sys/devices/system/edac/pci/check_pci_errors Disable:: - echo "0" >/sys/devices/system/edac/pci/check_pci_parity + echo "0" >/sys/devices/system/edac/pci/check_pci_errors - ``pci_parity_count`` - Parity Count @@ -725,15 +725,15 @@ Module parameters module/kernel parameter:: - edac_panic_on_pci_pe=[0|1] + edac_pci_panic_on_pe=[0|1] Enable:: - echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe + echo "1" > /sys/module/edac_core/parameters/edac_pci_panic_on_pe Disable:: - echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe + echo "0" > /sys/module/edac_core/parameters/edac_pci_panic_on_pe diff --git a/Documentation/admin-guide/binfmt-misc.rst b/Documentation/admin-guide/binfmt-misc.rst index d26b63a27c259b..9e84b877d06dea 100644 --- a/Documentation/admin-guide/binfmt-misc.rst +++ b/Documentation/admin-guide/binfmt-misc.rst @@ -19,6 +19,9 @@ To actually register a new binary type, you have to set up a string looking like ``:name:type:offset:magic:mask:interpreter:flags`` (where you can choose the ``:`` upon your needs) and echo it to ``/proc/sys/fs/binfmt_misc/register``. +The first character of the string is its field delimiter and can be any +ASCII punctuation character other than the backslash ``\``. + Here is what the fields mean: - ``name`` diff --git a/Documentation/admin-guide/cpu-isolation.rst b/Documentation/admin-guide/cpu-isolation.rst index 8c65d03fd28c2a..b9f13a92867343 100644 --- a/Documentation/admin-guide/cpu-isolation.rst +++ b/Documentation/admin-guide/cpu-isolation.rst @@ -353,5 +353,5 @@ Some tools may also be useful for higher level analysis: latency and noise in the system. For example Documentation/tools/rtla/rtla-osnoise.rst runs a kernel tracer that analyzes and output a summary of the noises. -- dynticks-testing does something similar to rtla-osnoise but in userspace. It is available - at git://git.kernel.org/pub/scm/linux/kernel/git/frederic/dynticks-testing.git +- cpunoise does something similar to rtla-osnoise but in userspace. It is available + at https://git.kernel.org/pub/scm/linux/kernel/git/frederic/cpunoise.git diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index e75344f4e0cdef..902daec557f052 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -501,13 +501,6 @@ Kernel parameters disable Disable amd-pstate preferred core. - amd_dynamic_epp= - [X86] - disable - Disable amd-pstate dynamic EPP. - enable - Enable amd-pstate dynamic EPP. - amijoy.map= [HW,JOY] Amiga joystick support Map of devices attached to JOY0DAT and JOY1DAT Format: , @@ -575,7 +568,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension support @@ -594,8 +588,6 @@ Kernel parameters ataflop= [HW,M68k] - atarimouse= [HW,MOUSE] Atari Mouse - atkbd.extra= [HW] Enable extra LEDs and keys on IBM RapidAccess, EzKey and similar keyboards @@ -629,13 +621,6 @@ Kernel parameters Format: (must be >=0) Default: 64 - bau= [X86_UV] Enable the BAU on SGI UV. The default - behavior is to disable the BAU (i.e. bau=0). - Format: { "0" | "1" } - 0 - Disable the BAU. - 1 - Enable the BAU. - unset - Disable the BAU. - bdev_allow_write_mounted= Format: Control the ability to open a mounted block device @@ -1590,14 +1575,6 @@ Kernel parameters PCI device even when its classcode is not of the UART class. - edac_report= [HW,EDAC] Control how to report EDAC event - Format: {"on" | "off" | "force"} - on: enable EDAC to report H/W event. May be overridden - by other higher priority error reporting module. - off: disable H/W event reporting through EDAC. - force: enforce the use of EDAC to report H/W event. - default: on. - edd= [EDD] Format: {"off" | "on" | "skip[mbr]"} @@ -1667,12 +1644,6 @@ Kernel parameters to discrete, to make X server driver able to add WB entry later. This parameter enables that. - enable_timer_pin_1 [X86] - Enable PIN 1 of APIC timer - Can be useful to work around chipset bugs - (in particular on some ATI chipsets). - The kernel tries to set a reasonable default. - enforcing= [SELINUX] Set initial enforcing status. Format: {"0" | "1"} See security/selinux/Kconfig help text. @@ -3969,8 +3940,8 @@ Kernel parameters like Grub2, otherwise '$' and the following number will be eaten. - memmap=nn[KMG]!ss[KMG,EARLY] - [KNL,X86] Mark specific memory as protected. + memmap=nn[KMG]!ss[KMG] + [KNL,X86,EARLY] Mark specific memory as protected. Region of memory to be used, from ss to ss+nn. The memory region may be marked as e820 type 12 (0xc) and is NVDIMM or ADR memory. @@ -4031,7 +4002,7 @@ Kernel parameters poison checking when freeing elements back to the pool. Useful for debugging mempool corruption. - memtest= [KNL,X86,ARM,M68K,PPC,RISCV,EARLY] Enable memtest + memtest= [KNL,ALPHA,X86,ARM,M68K,PPC,RISCV,EARLY] Enable memtest Format: default : 0 Specifies the number of memtest passes to be @@ -4076,27 +4047,6 @@ Kernel parameters Enable or disable the microcode minimal revision enforcement for the runtime microcode loader. - mini2440= [ARM,HW,KNL] - Format:[0..2][b][c][t] - Default: "0tb" - MINI2440 configuration specification: - 0 - The attached screen is the 3.5" TFT - 1 - The attached screen is the 7" TFT - 2 - The VGA Shield is attached (1024x768) - Leaving out the screen size parameter will not load - the TFT driver, and the framebuffer will be left - unconfigured. - b - Enable backlight. The TFT backlight pin will be - linked to the kernel VESA blanking code and a GPIO - LED. This parameter is not necessary when using the - VGA shield. - c - Enable the s3c camera interface. - t - Reserved for enabling touchscreen support. The - touchscreen support is not enabled in the mainstream - kernel as of 2.6.30, a preliminary port can be found - in the "bleeding edge" mini2440 support kernel at - https://repo.or.cz/w/linux-2.6/mini2440.git - mitigations= [X86,PPC,S390,ARM64,EARLY] Control optional mitigations for CPU vulnerabilities. This is a set of curated, @@ -4281,11 +4231,11 @@ Kernel parameters Enable printing debug information related to MTRR registers at boot time. - mtrr_chunk_size=nn[KMG,X86,EARLY] + mtrr_chunk_size=nn[KMG] [X86,EARLY] used for mtrr cleanup. It is largest continuous chunk that could hold holes aka. UC entries. - mtrr_gran_size=nn[KMG,X86,EARLY] + mtrr_gran_size=nn[KMG] [X86,EARLY] Used for mtrr cleanup. It is granularity of mtrr block. Default is 1. Large value could prevent small alignment from @@ -4598,9 +4548,6 @@ Kernel parameters nomce [X86-32] Disable Machine Check Exception - nomfgpt [X86-32] Disable Multi-Function General Purpose - Timer usage (for AMD Geode machines). - nomodeset Disable kernel modesetting. Most systems' firmware sets up a display mode and provides framebuffer memory for output. With nomodeset, DRM and fbdev drivers will @@ -4813,11 +4760,6 @@ Kernel parameters waiting for the ACK, so if this is set too high interrupts *may* be lost! - omap_mux= [OMAP] Override bootloader pin multiplexing. - Format: ... - For example, to override I2C bus2: - omap_mux=i2c2_scl.i2c2_scl=0x100,i2c2_sda.i2c2_sda=0x100 - onenand.bdry= [HW,MTD] Flex-OneNAND Boundary Configuration Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock] @@ -5035,6 +4977,9 @@ Kernel parameters the specified number of seconds. This is to be used if your oopses keep scrolling off the screen. + pax_softmode= + Enables OpenPaX soft mode if set to a non-zero value. + pci=option[,option...] [PCI,EARLY] various PCI subsystem options. Some options herein operate on a specific device @@ -5327,8 +5272,6 @@ Kernel parameters nomsi Do not use MSI for native PCIe PME signaling (this makes all PCIe root ports use INTx for all services). - pcmv= [HW,PCMCIA] BadgePAD 4 - pd_ignore_unused [PM] Keep all power-domains already enabled by bootloader on, @@ -5454,8 +5397,8 @@ Kernel parameters lazy - Scheduler controlled. Similar to full but instead of preempting the task immediately, the task gets one HZ tick time to yield itself before the - preemption will be forced. One preemption is when the - task returns to user space. + preemption will be forced. One such preemption + point is when the task returns to user space. print-fatal-signals= [KNL] debug: print fatal signals @@ -5663,9 +5606,12 @@ Kernel parameters there is an ongoing too-long CSD-lock wait. rcutree.do_rcu_barrier= [KNL] - Request a call to rcu_barrier(). This is - throttled so that userspace tests can safely - hammer on the sysfs variable if they so choose. + Wait for deferred kfree_rcu() frees and ordinary + call_rcu() callbacks queued before this request to + complete. This does not prevent new work from being + queued concurrently. Requests are throttled so that + userspace tests can safely hammer on the sysfs + variable if they so choose. If triggered before the RCU grace-period machinery is fully active, this will error out with EAGAIN. @@ -6137,6 +6083,13 @@ Kernel parameters stress RCU, they don't participate in the actual test, hence the "fake". + rcutorture.nmi_calls= [KNL] + Enable issuing RCU callbacks from an NMI, on the + RCU flavors that support it, to exercise the + any-context callback path. Requires + CONFIG_PERF_EVENTS and a hardware PMU; without + both, nothing is issued. Defaults to enabled. + rcutorture.nocbs_nthreads= [KNL] Set number of RCU callback-offload togglers. Zero (the default) disables toggling. @@ -6366,6 +6319,13 @@ Kernel parameters period to instead use normal non-expedited grace-period processing. + rcupdate.rcu_task_collapse_debug= [KNL] + Enable debugging prints that record when RCU Tasks + and RCU Tasks Trace expand to per-CPU callback + queuing and collapse back to CPU-0 queuing. + This is default-disabled due to the fact that + some workloads can make it quite noisy. + rcupdate.rcu_task_collapse_lim= [KNL] Set the maximum number of callbacks present at the beginning of a grace period that allows @@ -6590,7 +6550,7 @@ Kernel parameters them. If is less than 0x10000, the region is assumed to be I/O ports; otherwise it is memory. - reserve_mem= [RAM] + reserve_mem= [MM] Format: nn[KMG]::

, T); diff --git a/rust/pin-init/src/lib.rs b/rust/pin-init/src/lib.rs index f1463be9479d67..9ffa764343105a 100644 --- a/rust/pin-init/src/lib.rs +++ b/rust/pin-init/src/lib.rs @@ -304,6 +304,9 @@ pub use alloc::InPlaceInit; /// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`, /// then `#[pin]` directs the type of initializer that is required. /// +/// Tuple structs are supported as well. Their fields have no names, so the generated projection +/// is a tuple struct too and its fields are accessed by index. +/// /// If your `struct` implements `Drop`, then you need to add `PinnedDrop` as arguments to this /// macro, and change your `Drop` implementation to `PinnedDrop` annotated with /// `#[`[`macro@pinned_drop`]`]`, since dropping pinned values requires extra care. @@ -327,6 +330,26 @@ pub use alloc::InPlaceInit; /// } /// ``` /// +/// The same as a tuple struct, projected by index: +/// +/// ``` +/// # #![feature(allocator_api)] +/// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*; +/// use core::pin::Pin; +/// use pin_init::pin_data; +/// +/// enum Command { +/// /* ... */ +/// } +/// +/// #[pin_data] +/// struct DriverData(#[pin] CMutex>, Box<[u8; 1024 * 1024]>); +/// +/// fn queue(data: Pin<&mut DriverData>) -> Pin<&mut CMutex>> { +/// data.project().0 +/// } +/// ``` +/// /// ``` /// # #![feature(allocator_api)] /// # #[path = "../examples/mutex.rs"] mod mutex; use mutex::*; @@ -569,7 +592,7 @@ macro_rules! stack_try_pin_init { }; } -/// Construct an in-place, fallible pinned initializer for `struct`s. +/// Construct an in-place, fallible pinned initializer for structs, including tuple structs. /// /// The error type defaults to [`Infallible`]; if you need a different one, write `? Error` at the /// end, after the struct initializer. @@ -603,6 +626,42 @@ macro_rules! stack_try_pin_init { /// # Box::pin_init(demo()).unwrap(); /// ``` /// +/// The fields of a tuple struct are addressed by their index: +/// +/// ```rust +/// # use pin_init::*; +/// # use core::pin::Pin; +/// #[pin_data] +/// struct Pair(usize, Bar); +/// +/// #[pin_data] +/// struct Bar { +/// x: u32, +/// } +/// +/// # fn demo() -> impl PinInit { +/// let initializer = pin_init!(Pair { +/// 0: 42, +/// 1 <- Bar { x: 64 }, +/// }); +/// # initializer } +/// # Box::pin_init(demo()).unwrap(); +/// ``` +/// +/// A tuple struct whose fields are all set to a value can also be written like a call to its +/// constructor: +/// +/// ```rust +/// # use pin_init::*; +/// #[pin_data] +/// struct Pair(usize, usize); +/// +/// # fn demo() -> impl PinInit { +/// let initializer = pin_init!(Pair(42, 64)); +/// # initializer } +/// # Box::pin_init(demo()).unwrap(); +/// ``` +/// /// Arbitrary Rust expressions can be used to set the value of a variable. /// /// The fields are initialized in the order that they appear in the initializer. So it is possible @@ -721,9 +780,13 @@ macro_rules! stack_try_pin_init { /// /// # Syntax /// -/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with -/// the following modifications is expected: +/// As already mentioned in the examples above, inside of `pin_init!` a struct initializer with the +/// following modifications is expected: /// - Fields that you want to initialize in-place have to use `<-` instead of `:`. +/// - Tuple struct fields are named by their index, as in `0: value` or `0 <- initializer`. They +/// are not exposed by a `let` binding, since they have no name to bind. +/// - A tuple struct can also be initialized with constructor syntax, as in `Type(value, value)`. +/// Since its arguments are not named, they cannot use `<-`; write them out by index instead. /// - You can use `_: { /* run any user-code here */ },` anywhere where you can place fields in /// order to run arbitrary code. /// - In front of the initializer you can write `&this in` to have access to a [`NonNull`] @@ -762,7 +825,7 @@ macro_rules! stack_try_pin_init { /// [`NonNull`]: core::ptr::NonNull pub use pin_init_internal::pin_init; -/// Construct an in-place, fallible initializer for `struct`s. +/// Construct an in-place, fallible initializer for structs, including tuple structs. /// /// This macro defaults the error to [`Infallible`]; if you need a different one, write `? Error` /// at the end, after the struct initializer. @@ -857,8 +920,10 @@ macro_rules! assert_pinned { ($ty:ty, $field:ident, $field_ty:ty, inline) => { // SAFETY: This code is unreachable. let _ = move |ptr: *mut $ty| unsafe { - let data = <$ty as $crate::__internal::HasPinData>::__pin_data(); + let data = <$ty as $crate::__internal::HasInitData>::__init_data(); + let data = $crate::__internal::HasPinData::__pin_data(data); _ = data + .__with_lt() .$field(ptr) .init($crate::__internal::AlwaysFail::<$field_ty>::new()); }; @@ -902,6 +967,9 @@ macro_rules! assert_pinned { #[cfg_attr(not(kernel), doc = "[`Arc`]: alloc::alloc::sync::Arc")] #[cfg_attr(not(kernel), doc = "[`Box`]: alloc::alloc::boxed::Box")] #[must_use = "An initializer must be used in order to create its value."] +#[diagnostic::on_unimplemented( + message = "`{Self}` cannot be used to initialize `{T}` with error `{E}`" +)] pub unsafe trait PinInit: Sized { /// Alias of [`PinInit::__init`]. /// @@ -1036,6 +1104,11 @@ where #[cfg_attr(not(kernel), doc = "[`Arc`]: alloc::alloc::sync::Arc")] #[cfg_attr(not(kernel), doc = "[`Box`]: alloc::alloc::boxed::Box")] #[must_use = "An initializer must be used in order to create its value."] +#[diagnostic::on_unimplemented( + message = "`{Self}` cannot be used to movably initialize `{T}` with error `{E}`", + note = "if your type implements `PinInit` but not `Init`, \ + you might be forgetting a `#[pin]` annotation on fields" +)] pub unsafe trait Init: PinInit { /// First initializes the value using `self` then calls the function `f` with the initialized /// value. diff --git a/rust/proc-macro2/README.md b/rust/proc-macro2/README.md index af044fee4f59c5..519d01112a022b 100644 --- a/rust/proc-macro2/README.md +++ b/rust/proc-macro2/README.md @@ -4,7 +4,8 @@ These source files come from the Rust `proc-macro2` crate, version 1.0.101 (released 2025-08-16), hosted in the repository, licensed under "Apache-2.0 OR MIT" and only modified to add the SPDX license -identifiers and to remove the `unicode-ident` dependency. +identifiers, to remove the `unicode-ident` dependency, and to build +with 1.85 with nightly features enabled. For copyright details, please see: diff --git a/rust/proc-macro2/probe/proc_macro_span.rs b/rust/proc-macro2/probe/proc_macro_span.rs index 892a7eb3e5a05e..765bd47dbecd75 100644 --- a/rust/proc-macro2/probe/proc_macro_span.rs +++ b/rust/proc-macro2/probe/proc_macro_span.rs @@ -10,7 +10,6 @@ extern crate proc_macro; use core::ops::{Range, RangeBounds}; use proc_macro::{Literal, Span}; -use std::path::PathBuf; pub fn byte_range(this: &Span) -> Range { this.byte_range() @@ -32,14 +31,6 @@ pub fn column(this: &Span) -> usize { this.column() } -pub fn file(this: &Span) -> String { - this.file() -} - -pub fn local_file(this: &Span) -> Option { - this.local_file() -} - pub fn join(this: &Span, other: Span) -> Option { this.join(other) } diff --git a/rust/zerocopy-derive/README.md b/rust/zerocopy-derive/README.md index d62c7980434248..fab3779b9ed22f 100644 --- a/rust/zerocopy-derive/README.md +++ b/rust/zerocopy-derive/README.md @@ -1,14 +1,14 @@ # `zerocopy-derive` -These source files come from the Rust `zerocopy-derive` crate, version v0.8.54 -(released 2026-07-08), hosted in the +These source files come from the Rust `zerocopy-derive` crate, version v0.8.62 +(released 2026-10-08), hosted in the repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only modified to tweak the SPDX license identifiers and to remove the generation of non-ASCII identifiers. For copyright details, please see: - https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1 - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT + https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1 + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT diff --git a/rust/zerocopy-derive/derive/from_bytes.rs b/rust/zerocopy-derive/derive/from_bytes.rs index 66d820f6ad4cde..7fe505a86457a8 100644 --- a/rust/zerocopy-derive/derive/from_bytes.rs +++ b/rust/zerocopy-derive/derive/from_bytes.rs @@ -89,6 +89,12 @@ pub(crate) fn find_zero_variant(enm: &DataEnum) -> Result { Err(has_unknown_discriminants) } pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result { + if let Some(span) = ctx.invariant_span { + return ctx.error_or_skip(Error::new( + span, + "cannot derive `FromZeros` for a type with invariants", + )); + } let try_from_bytes = derive_try_from_bytes(ctx, top_level)?; let from_zeros = match &ctx.ast.data { Data::Struct(strct) => derive_from_zeros_struct(ctx, strct), @@ -98,6 +104,12 @@ pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result Result { + if let Some(span) = ctx.invariant_span { + return ctx.error_or_skip(Error::new( + span, + "cannot derive `FromBytes` for a type with invariants", + )); + } let from_zeros = derive_from_zeros(ctx, top_level)?; let from_bytes = match &ctx.ast.data { Data::Struct(strct) => derive_from_bytes_struct(ctx, strct), diff --git a/rust/zerocopy-derive/derive/into_bytes.rs b/rust/zerocopy-derive/derive/into_bytes.rs index 0103a78d087f7c..55c53de6b8c453 100644 --- a/rust/zerocopy-derive/derive/into_bytes.rs +++ b/rust/zerocopy-derive/derive/into_bytes.rs @@ -2,7 +2,7 @@ // use proc_macro2::{Span, TokenStream}; use quote::quote; -use syn::{Data, DataEnum, DataStruct, DataUnion, Error, Type}; +use syn::{parse_quote, Data, DataEnum, DataStruct, DataUnion, Error, Ident, Type, WherePredicate}; use crate::{ repr::{EnumRepr, StructUnionRepr}, @@ -18,6 +18,74 @@ pub(crate) fn derive_into_bytes(ctx: &Ctx, _top_level: Trait) -> Result derive_into_bytes_union(ctx, unn), } } + +/// If every field is exactly `T`, `[T; _]`, or a final `[T]` for the same type +/// parameter `T`, returns the bounds required to prove this to rustc. +fn homogeneous_field_bounds(ctx: &Ctx, strct: &DataStruct) -> Option> { + fn strip_parens_and_groups(mut ty: &Type) -> &Type { + loop { + ty = match ty { + Type::Group(group) => &group.elem, + Type::Paren(paren) => &paren.elem, + ty => return ty, + }; + } + } + + fn type_is_parameter(ty: &Type, parameter: &Ident) -> bool { + match strip_parens_and_groups(ty) { + Type::Path(path) => path.qself.is_none() && path.path.is_ident(parameter), + _ => false, + } + } + + let fields = strct.fields(); + ctx.ast.generics.type_params().find_map(|parameter| { + let parameter = ¶meter.ident; + let elements = fields + .iter() + .enumerate() + .map(|(index, (_, _, ty))| match strip_parens_and_groups(ty) { + Type::Array(array) => Some(&*array.elem), + Type::Slice(slice) if index + 1 == fields.len() => Some(&*slice.elem), + Type::Slice(_) => None, + ty => Some(ty), + }) + .collect::>>()?; + + if !elements.iter().all(|element| type_is_parameter(element, parameter)) { + return None; + } + + let zerocopy_crate = &ctx.zerocopy_crate; + let mut bounds = fields + .iter() + .map(|(_, _, ty)| parse_quote!(#ty: #zerocopy_crate::IntoBytes)) + .collect::>(); + + // `Ident` equality is only a syntactic prefilter above. In + // macro-generated input, two identifiers can both print as `T` and + // compare equal here while having different hygiene contexts. For + // example, a call-site `T` could resolve to a `u8` type alias while a + // def-site `T` resolves to this struct's type parameter. Instantiating + // that parameter with `u16` would make the fields actually be `u8` and + // `u16`; treating them as homogeneous could overlook padding and cause + // us to emit an unsound `IntoBytes` impl. + // + // Quote every element type from its original field AST so that its + // hygiene context is retained, then use `Identity` to have rustc verify + // that it resolves to `parameter`. Do not deduplicate these predicates: + // identically-printed types may have different hygiene contexts and + // must each be checked by rustc. + bounds.extend(elements.into_iter().map(|element| { + parse_quote! { + #element: #zerocopy_crate::util::macro_util::Identity + } + })); + Some(bounds) + }) +} + fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result { let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?; @@ -25,8 +93,12 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result Result Result Result Result TokenStream { + let ctx = Ctx::try_from_derive_input(input).unwrap(); + derive_into_bytes(&ctx, Trait::IntoBytes).unwrap() + } + + #[test] + fn raw_repr_matches_ordinary_repr() { + let ordinary = derive(parse_quote! { + #[repr(C)] + #[repr(align(8))] + struct Response { + status: u8, + } + }); + let raw = derive(parse_quote! { + #[repr(C)] + #[r#repr(r#align(8))] + struct Response { + status: u8, + } + }); + + let ordinary = ordinary.to_string(); + let raw = raw.to_string(); + + // In particular, both expansions contain the padding check which + // prevents the resulting type from implementing `IntoBytes`. + assert!(ordinary.contains("PaddingFree")); + assert_eq!(ordinary, raw); + } +} diff --git a/rust/zerocopy-derive/derive/known_layout.rs b/rust/zerocopy-derive/derive/known_layout.rs index d0c4cecfff151e..d4f6ecff573006 100644 --- a/rust/zerocopy-derive/derive/known_layout.rs +++ b/rust/zerocopy-derive/derive/known_layout.rs @@ -81,7 +81,7 @@ fn derive_known_layout_for_repr_c_struct<'a>( #[inline(always)] fn pointer_to_metadata(ptr: *mut Self) -> ::PointerMetadata { - <#trailing_field_ty>::pointer_to_metadata(ptr as *mut _) + <#trailing_field_ty as #zerocopy_crate::KnownLayout>::pointer_to_metadata(ptr as *mut _) } } }; @@ -144,6 +144,7 @@ fn derive_known_layout_for_repr_c_struct<'a>( // Define the collection of type-level field handles. let field_defs = field_indices.iter().zip(fields).map(|(idx, (vis, _, _))| { quote! { + #[allow(missing_debug_implementations)] #vis struct #idx; } }); @@ -200,6 +201,7 @@ fn derive_known_layout_for_repr_c_struct<'a>( // `#ty`, not `__ZerocopyKnownLayoutMaybeUninit` (see #2116). #repr #[doc(hidden)] + #[allow(missing_debug_implementations)] #vis struct __ZerocopyKnownLayoutMaybeUninit<#params> ( #(#core::mem::MaybeUninit< <#ident #ty_generics as diff --git a/rust/zerocopy-derive/derive/mod.rs b/rust/zerocopy-derive/derive/mod.rs index b3839fcf73c972..4ccb71807ee750 100644 --- a/rust/zerocopy-derive/derive/mod.rs +++ b/rust/zerocopy-derive/derive/mod.rs @@ -3,6 +3,7 @@ pub mod from_bytes; pub mod into_bytes; pub mod known_layout; +pub mod project; pub mod try_from_bytes; pub mod unaligned; @@ -102,13 +103,6 @@ pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result Result::Elem); + let elem = + if repr.get_packed().is_some() { quote!(#zerocopy_crate::Unalign<#elem>) } else { elem }; + + // SAFETY: The checks above require `repr(C)` or `repr(transparent)`, and + // `FieldBounds::TRAILING_SELF` requires the trailing field to be `SplitAt`. + // Its `Elem` therefore has the size, bit validity, and `UnsafeCell` + // coverage of the actual trailing element. For an unpacked struct, the + // trailing field is aligned, so its `SplitAt` alignment guarantee applies. + // For a packed struct, wrapping its `Elem` in `Unalign` preserves those + // representation properties and gives alignment 1. Thus the trailing + // slice can be accessed as `[#elem]` even if packing misaligns its field. + // Packing does not change the field's internal layout [1], so its element + // size and byte offsets are preserved in either case. + // + // [1] Per https://doc.rust-lang.org/1.93.0/reference/type-layout.html#layout.repr.alignment.intro: + // + // `packed` may also alter the padding between fields (although it will + // not alter the padding inside of any field). Ok(ImplBlockBuilder::new(ctx, &ctx.ast.data, Trait::SplitAt, FieldBounds::TRAILING_SELF) .inner_extras(quote! { - type Elem = <#trailing_field as #zerocopy_crate::SplitAt>::Elem; + type Elem = #elem; }) .build()) } diff --git a/rust/zerocopy-derive/derive/project.rs b/rust/zerocopy-derive/derive/project.rs new file mode 100644 index 00000000000000..1cfde70624a7b7 --- /dev/null +++ b/rust/zerocopy-derive/derive/project.rs @@ -0,0 +1,569 @@ +// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT +// +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use proc_macro2::TokenStream; +use quote::quote; +use syn::{ + parse_quote, spanned::Spanned as _, Data, DataEnum, DeriveInput, Error, Expr, Fields, Ident, + Index, Type, +}; + +use crate::{ + repr::{EnumRepr, StructUnionRepr}, + util::{ + const_block, generate_tag_enum, Client, Ctx, DataExt, FieldBounds, FieldProjection, + ImplBlockBuilder, Trait, + }, +}; + +pub(crate) fn tag_ident(variant: &Ident) -> Ident { + ident!(("___ZEROCOPY_TAG_{}", variant), variant.span()) +} + +pub(crate) fn variant_struct_ident(variant: &Ident) -> Ident { + ident!(("___ZerocopyVariantStruct_{}", variant), variant.span()) +} + +pub(crate) fn variants_union_field_ident(variant: &Ident) -> Ident { + ident!(("__field_{}", variant), variant.span()) +} + +/// Generates a constant for the tag associated with each variant of the enum. +/// When we match on the enum's tag, each arm matches one of these constants. We +/// have to use constants here because: +/// +/// - The type that we're matching on is not the type of the tag, it's an +/// integer of the same size as the tag type and with the same bit patterns. +/// - We can't read the enum tag as an enum because the bytes may not represent +/// a valid variant. +/// - Patterns do not currently support const expressions, so we have to assign +/// these constants to names rather than use them inline in the `match` +/// statement. +pub(crate) fn generate_tag_consts(data: &DataEnum) -> TokenStream { + let tags = data.variants.iter().map(|variant| { + let variant_ident = &variant.ident; + let tag = tag_ident(variant_ident); + + quote! { + // This casts the enum variant to its discriminant, and then + // converts the discriminant to the target integral type via a + // numeric cast [1]. + // + // Because these are the same size, this is defined to be a no-op + // and therefore is a lossless conversion [2]. + // + // [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast: + // + // Casts an enum to its discriminant. + // + // [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast: + // + // Casting between two integers of the same size (e.g. i32 -> u32) + // is a no-op. + const #tag: ___ZerocopyTagPrimitive = + ___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive; + } + }); + + quote! { + #(#tags)* + } +} + +#[derive(Copy, Clone, Eq, PartialEq)] +enum Validity { + Uninit, + Initialized, + Safe, +} + +impl Validity { + fn as_type(self, ctx: &Ctx) -> Type { + let zerocopy_crate = &ctx.zerocopy_crate; + match self { + Validity::Uninit => parse_quote!(#zerocopy_crate::invariant::Uninit), + Validity::Initialized => { + parse_quote!(#zerocopy_crate::invariant::Initialized) + } + Validity::Safe => parse_quote!(#zerocopy_crate::invariant::Safe), + } + } + + /// Computes the output validity of projecting a struct or union field, or + /// returns `None` if that projection must not be generated. + /// + /// Struct fields preserve all three validity invariants. Union fields + /// preserve `Uninit` and `Initialized`. No projection is generated from a + /// `Safe` union. + fn output_validity_for_struct_or_union(self, data: &Data) -> Option { + match (data, self) { + (Data::Struct(_), validity) => Some(validity), + (Data::Union(_), validity @ (Validity::Uninit | Validity::Initialized)) => { + Some(validity) + } + (Data::Union(_), Validity::Safe) => None, + (Data::Enum(_), _) => unreachable!(), + } + } +} + +/// Returns the projection variant ID for the struct or union described by +/// `ctx`. +/// +/// This intentionally does not handle enums: an enum field's variant ID +/// identifies its source variant rather than the representation of its +/// containing type. +pub(crate) fn struct_union_variant_id(ctx: &Ctx) -> Box { + let zerocopy_crate = &ctx.zerocopy_crate; + match &ctx.ast.data { + Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }), + Data::Union(_) => { + if StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap() { + parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID }) + } else { + parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID }) + } + } + Data::Enum(_) => unreachable!("struct_union_variant_id called for enum"), + } +} + +/// Generates field projection implementations for a struct or union. +/// +/// `repr(C)` unions use `REPR_C_UNION_VARIANT_ID`, so their projections +/// additionally implement `pointer::cast::Cast`. Other unions use +/// `UNION_VARIANT_ID`. +pub(crate) fn derive_projection_struct_union( + ctx: &Ctx, + data: &dyn DataExt, + client: Client, +) -> TokenStream { + let fields = ctx.ast.data.fields(); + if fields.is_empty() { + return quote! {}; + } + + let field_tokens = fields.iter().map(|(vis, ident, _)| { + let ident = ident!(("__z{}", ident), ident.span()); + quote! { + #vis enum #ident {} + } + }); + + let zerocopy_crate = &ctx.zerocopy_crate; + let variant_id = struct_union_variant_id(ctx); + + let core = ctx.core_path(); + let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag { client }, FieldBounds::None) + .inner_extras(quote! { + type Tag = (); + type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit; + }) + .build(); + + let has_fields = fields.iter().map(move |(_, ident, ty)| { + let field_token = ident!(("__z{}", ident), ident.span()); + let projection = FieldProjection { + variant_id: variant_id.clone(), + field: parse_quote!(#field_token), + field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }), + }; + let has_field_trait = Trait::HasField { client, projection: projection.clone() }; + let has_field_path = has_field_trait.crate_path(ctx); + let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None) + .inner_extras(quote! { + type Type = #ty; + + #[inline(always)] + fn project( + slf: #zerocopy_crate::pointer::PtrInner<'_, Self>, + ) -> *mut ::Type { + let slf = slf.as_ptr(); + // SAFETY: By invariant on `PtrInner`, `slf` is a non-null + // pointer whose referent is zero-sized or lives in a valid + // allocation. Since `#ident` is a struct or union field of + // `Self`, this projection preserves or shrinks the referent + // size, and so the resulting referent also fits in the same + // allocation. + unsafe { #core::ptr::addr_of_mut!((*slf).#ident) } + } + }) + .build(); + + let derive_project_field = |input_validity: Validity| { + let output_validity = + match input_validity.output_validity_for_struct_or_union(&ctx.ast.data) { + Some(output_validity) => output_validity.as_type(ctx), + None => return TokenStream::new(), + }; + let input_validity = input_validity.as_type(ctx); + let projection_preserves_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs) + .map(|repr| repr.get_packed().is_none()) + .unwrap(); + let output_alignment = if projection_preserves_alignment { + quote! { ___ZcAlignment } + } else { + quote! { #zerocopy_crate::invariant::Unaligned } + }; + ImplBlockBuilder::new( + ctx, + data, + Trait::ProjectField { + client, + projection: projection.clone(), + invariants: parse_quote!((___ZcAliasing, ___ZcAlignment, #input_validity)), + }, + FieldBounds::None, + ) + .param_extras(vec![ + parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing), + parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment), + ]) + .inner_extras(quote! { + // SAFETY: Struct and union projections do not depend on the value of + // the referent, and are therefore infallible. + type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible; + + // SAFETY: Projection preserves aliasing. It also preserves + // alignment unless the containing type is packed. `Validity` + // determines whether a projection is emitted and, if so, + // computes its output validity: struct fields preserve validity; + // union fields preserve `Uninit` and `Initialized`, and no + // projection is emitted for `Safe` union input. + type Invariants = (___ZcAliasing, #output_alignment, #output_validity); + }) + .build() + }; + + let project_uninit = derive_project_field(Validity::Uninit); + let project_initialized = derive_project_field(Validity::Initialized); + let project_safe = derive_project_field(Validity::Safe); + + quote! { + #has_field + #project_uninit + #project_initialized + #project_safe + } + }); + + const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some)) +} + +pub(crate) fn derive_enum( + ctx: &Ctx, + data: &DataEnum, + client: Client, +) -> Result { + // With no fields, there are no `HasField` or `ProjectField` impls to emit, + // and therefore no representation requirement to enforce. + if data.fields().is_empty() { + return Ok(TokenStream::new()); + } + + let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?; + let (outer_tag_type, inner_tag_type) = if repr.is_c() { + (quote! { ___ZerocopyTag }, quote! { () }) + } else if repr.is_primitive() { + (quote! { () }, quote! { ___ZerocopyTag }) + } else { + return Err(Error::new( + ctx.ast.span(), + "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout", + )); + }; + + let zerocopy_crate = &ctx.zerocopy_crate; + let core = &ctx.core_path(); + let client_path = &client.crate_path(ctx); + let has_tag_trait = Trait::HasTag { client }; + let has_tag_path = &has_tag_trait.crate_path(ctx); + let infallible = + quote! { #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible }; + let tag_enum = generate_tag_enum(ctx, &repr, data); + let tag_consts = generate_tag_consts(data); + + let generics = &ctx.ast.generics; + let (impl_generics, ref ty_generics, where_clause) = generics.split_for_impl(); + + // Synthesize a `repr(C)` struct for each fieldful variant. We must do this + // because Rust does not provide an `addr_of!` that works on enums. + let variant_structs = data.variants.iter().filter_map(move |variant| { + if matches!(variant.fields, Fields::Unit) { + return None; + } + + let ident = variant_struct_ident(&variant.ident); + let field_types = variant.fields.iter().map(|field| &field.ty); + let enum_name = &ctx.ast.ident; + let variant_struct: DeriveInput = parse_quote! { + #[repr(C)] + struct #ident #impl_generics ( + #core::mem::MaybeUninit<___ZerocopyInnerTag>, + #(#field_types,)* + #core::marker::PhantomData<#enum_name #ty_generics>, + ) #where_clause; + }; + + let projections = derive_projection_struct_union( + &ctx.with_input(&variant_struct), + &variant_struct.data, + client, + ); + + Some(quote! { + #variant_struct + #projections + }) + }); + + // Synthesize a `repr(C)` union encompassing all synthesized variants. + let variants_union = { + let fields = data.variants.iter().filter_map(|variant| { + if matches!(variant.fields, Fields::Unit) { + return None; + } + let field_name = variants_union_field_ident(&variant.ident); + let variant_struct = variant_struct_ident(&variant.ident); + Some(quote! { + #field_name: #core::mem::ManuallyDrop<#variant_struct #ty_generics>, + }) + }); + + let variants_union: DeriveInput = parse_quote! { + #[repr(C)] + union ___ZerocopyVariants #generics { + #(#fields)* + // A fieldless enum produces no variant structs, but a union must + // have at least one field. This unit does not affect `repr(C)` + // layout. + __nonempty: (), + } + }; + let projections = derive_projection_struct_union( + &ctx.with_input(&variants_union), + &variants_union.data, + client, + ); + + quote! { + #variants_union + #projections + } + }; + + // Synthesize a `repr(C)` struct that pairs the the tag with the synthesized + // variants union. In practice, the actual tag storage may be here, or + // inside the synthesized variant structs, depending on the `repr` of the + // enum. + let raw_enum: DeriveInput = parse_quote! { + #[repr(C)] + struct ___ZerocopyRawEnum #generics { + tag: ___ZerocopyOuterTag, + variants: ___ZerocopyVariants #ty_generics, + } + }; + + let raw_projections = + derive_projection_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data, client); + + let has_tag = ImplBlockBuilder::new(ctx, data, has_tag_trait, FieldBounds::None) + .inner_extras(quote! { + type Tag = ___ZerocopyTag; + type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized; + }) + .build(); + + let aliasing_param: &syn::GenericParam = + &parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing); + let reference_param: &syn::GenericParam = + &parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Reference); + let alignment_param: &syn::GenericParam = + &parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment); + + let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| { + let variant_ident = &variant.unwrap().ident; + let variants_union_field = variants_union_field_ident(variant_ident); + let infallible = &infallible; + + fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| { + // Rust does not presently support explicit visibility modifiers on + // enum fields. Keep this assertion so that a future language + // change cannot silently invalidate the visibility invariant. + assert!(matches!(vis, syn::Visibility::Inherited)); + let projection = FieldProjection { + variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }), + field: parse_quote!(()), + field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }), + }; + // The generated variant struct's first field is its inner tag. + let variant_struct_field_index = Index::from(idx + 1); + let has_field_trait = Trait::HasField { + client, + projection: projection.clone(), + }; + let has_field_path = has_field_trait.crate_path(ctx); + let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None) + .inner_extras(quote! { + type Type = #ty; + + #[inline(always)] + fn project( + slf: #zerocopy_crate::pointer::PtrInner<'_, Self>, + ) -> *mut ::Type { + use #zerocopy_crate::pointer::cast::{CastSized, Projection}; + + slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>() + .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>() + .project::<_, Projection<#client_path, _, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field) }>>() + .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>() + .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>() + .as_ptr() + } + }) + .build(); + + // SAFETY: `projection` is constructed from the current field and + // `variant_ident`, and that same `variant_ident` selects the + // variants union field through which `has_field` projects. Thus the + // `HasField` impl, `ProjectField` impls, and valid projection's tag + // check all identify the same enum field. `idx` is the current + // field's zero-based position, and the generated variant struct + // prepends its inner tag before reproducing the enum fields in + // source order, so `idx + 1` selects that same field. The assertion + // above enforces that `repr` is `repr(C)` or a primitive + // representation; `___ZerocopyRawEnum` and its nested types model + // exactly the representations specified by [1] and [2]. Those + // representations use `repr(C)` structs for variant fields, so field + // projection preserves alignment. Each `ProjectField` impl preserves + // the input validity. The `Uninit` and `Initialized` projections are + // infallible because those validity invariants do not depend on the + // enum's tag. The `Safe` projection first checks that the tag + // selects `variant_ident`; its `Reference` aliasing bound prevents + // mutation of the tag between that check and field projection. The + // corresponding `HasTag` impl projects to the raw representation's + // tag using `CastSized`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#reprc-enums-with-fields: + // + // The representation of a `repr(C)` enum with fields is a + // `repr(C)` struct with two fields [...]: a `repr(C)` version of + // the enum with all fields removed ("the tag") [and] a `repr(C)` + // union of `repr(C)` structs for the fields of each variant [...]. + // + // [2] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#primitive-representation-of-enums-with-fields: + // + // The representation of a primitive representation enum is a + // `repr(C)` union of `repr(C)` structs for each variant with a + // field. The first field of each struct in the union is [...] the + // tag and the remaining fields are the fields of that variant. + let derive_enum_project_field = |validity| { + let (error, is_projectable, aliasing_bound) = if validity == Validity::Safe { + assert!(!matches!(validity, Validity::Uninit | Validity::Initialized)); + ( + quote! { () }, + quote! { + #[inline(always)] + fn is_projectable( + tag: #zerocopy_crate::pointer::Ptr< + '_, + ::Tag, + ( + #zerocopy_crate::invariant::Shared, + ___ZcAlignment, + #zerocopy_crate::invariant::Safe, + ), + >, + ) -> #core::result::Result<(), ()> { + let tag = tag.read::<#zerocopy_crate::BecauseImmutable>(); + if tag == ___ZerocopyTag::#variant_ident { + #core::result::Result::Ok(()) + } else { + #core::result::Result::Err(()) + } + } + }, + reference_param.clone(), + ) + } else { + (infallible.clone(), quote! {}, aliasing_param.clone()) + }; + + let validity = validity.as_type(ctx); + let invariants: Box = + parse_quote!((___ZcAliasing, ___ZcAlignment, #validity)); + let project_field_trait = Trait::ProjectField { + client, + projection: projection.clone(), + invariants: invariants.clone(), + }; + + ImplBlockBuilder::new(ctx, data, project_field_trait, FieldBounds::None) + .param_extras(vec![aliasing_bound.clone(), alignment_param.clone()]) + .inner_extras(quote! { + type Error = #error; + type Invariants = #invariants; + #is_projectable + }) + .build() + }; + + let project_uninit = derive_enum_project_field(Validity::Uninit); + let project_initialized = derive_enum_project_field(Validity::Initialized); + let project_valid = derive_enum_project_field(Validity::Safe); + + quote! { + #has_field + #project_uninit + #project_initialized + #project_valid + } + }) + }); + + Ok(quote! { + #tag_enum + + type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag< + { #core::mem::size_of::<___ZerocopyTag>() }, + >; + + #tag_consts + + type ___ZerocopyOuterTag = #outer_tag_type; + type ___ZerocopyInnerTag = #inner_tag_type; + + #(#variant_structs)* + #variants_union + + #raw_enum + #raw_projections + + #has_tag + #(#has_fields)* + }) +} + +pub(crate) fn derive(ctx: &Ctx, _top_level: Trait) -> Result { + match &ctx.ast.data { + Data::Struct(strct) => { + Ok(derive_projection_struct_union(ctx, strct, Client::ProjectDerive)) + } + Data::Union(unn) => Ok(derive_projection_struct_union(ctx, unn, Client::ProjectDerive)), + Data::Enum(enm) => { + let projections = derive_enum(ctx, enm, Client::ProjectDerive)?; + if projections.is_empty() { + Ok(projections) + } else { + Ok(const_block([Some(projections)])) + } + } + } +} diff --git a/rust/zerocopy-derive/derive/try_from_bytes.rs b/rust/zerocopy-derive/derive/try_from_bytes.rs index 44f08332878641..ae0c796b91b166 100644 --- a/rust/zerocopy-derive/derive/try_from_bytes.rs +++ b/rust/zerocopy-derive/derive/try_from_bytes.rs @@ -1,442 +1,224 @@ // SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT // -use proc_macro2::TokenStream; -use quote::quote; -use syn::{ - parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error, - Expr, Fields, Ident, Index, Type, -}; +use proc_macro2::{Ident, Span, TokenStream}; +use quote::{quote, ToTokens as _}; +use syn::{spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, Error, Field}; use crate::{ - repr::{EnumRepr, StructUnionRepr}, + derive::project::{ + derive_enum, derive_projection_struct_union, generate_tag_consts, struct_union_variant_id, + tag_ident, + }, + repr::EnumRepr, util::{ - const_block, enum_size_from_repr, generate_tag_enum, Ctx, DataExt, FieldBounds, + enum_size_from_repr, generate_tag_enum, Client, Ctx, DataExt, FieldBounds, ImplBlockBuilder, Trait, TraitBound, }, }; -fn tag_ident(variant_ident: &Ident) -> Ident { - ident!(("___ZEROCOPY_TAG_{}", variant_ident), variant_ident.span()) -} - -/// Generates a constant for the tag associated with each variant of the enum. -/// When we match on the enum's tag, each arm matches one of these constants. We -/// have to use constants here because: -/// -/// - The type that we're matching on is not the type of the tag, it's an -/// integer of the same size as the tag type and with the same bit patterns. -/// - We can't read the enum tag as an enum because the bytes may not represent -/// a valid variant. -/// - Patterns do not currently support const expressions, so we have to assign -/// these constants to names rather than use them inline in the `match` -/// statement. -fn generate_tag_consts(data: &DataEnum) -> TokenStream { - let tags = data.variants.iter().map(|v| { - let variant_ident = &v.ident; - let tag_ident = tag_ident(variant_ident); - - quote! { - // This casts the enum variant to its discriminant, and then - // converts the discriminant to the target integral type via a - // numeric cast [1]. - // - // Because these are the same size, this is defined to be a no-op - // and therefore is a lossless conversion [2]. - // - // [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast: - // - // Casts an enum to its discriminant. - // - // [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast: - // - // Casting between two integers of the same size (e.g. i32 -> u32) - // is a no-op. - const #tag_ident: ___ZerocopyTagPrimitive = - ___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive; - } - }); - quote! { - #(#tags)* +fn candidate_ident(ctx: &Ctx) -> Ident { + // Choose a source-pointer name distinct from the field names. Mixed-site + // hygiene keeps it out of the user's invariant expressions. + let fields = ctx.ast.data.fields(); + let mut name = "candidate".to_owned(); + while fields.iter().any(|(_, field, _)| field.to_string().trim_start_matches("r#") == name) { + name.push('_'); } + Ident::new(&name, Span::mixed_site()) } -fn variant_struct_ident(variant_ident: &Ident) -> Ident { - ident!(("___ZerocopyVariantStruct_{}", variant_ident), variant_ident.span()) -} - -/// Generates variant structs for the given enum variant. -/// -/// These are structs associated with each variant of an enum. They are -/// `repr(C)` tuple structs with the same fields as the variant after a -/// `MaybeUninit<___ZerocopyInnerTag>`. -/// -/// In order to unify the generated types for `repr(C)` and `repr(int)` enums, -/// we use a "fused" representation with fields for both an inner tag and an -/// outer tag. Depending on the repr, we will set one of these tags to the tag -/// type and the other to `()`. This lets us generate the same code but put the -/// tags in different locations. -fn generate_variant_structs(ctx: &Ctx, data: &DataEnum) -> TokenStream { - let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl(); - - let enum_name = &ctx.ast.ident; +/// Generates validation of every field of a struct or enum variant, leaving +/// the validated pointers in scope for subsequent field invariants. +/// A union is checked by invoking this once per field. +fn derive_variant_is_safe<'a>( + ctx: &Ctx, + variant_id: &TokenStream, + fields: impl IntoIterator, +) -> Result { + let fields = fields.into_iter().collect::>(); + if fields.is_empty() { + return Ok(quote!(true)); + } - // All variant structs have a `PhantomData>` field because we - // don't know which generic parameters each variant will use, and unused - // generic parameters are a compile error. + let zerocopy_crate = &ctx.zerocopy_crate; let core = ctx.core_path(); - let phantom_ty = quote! { - #core::marker::PhantomData<#enum_name #ty_generics> - }; - - let variant_structs = data.variants.iter().filter_map(|variant| { - // We don't generate variant structs for unit variants because we only - // need to check the tag. This helps cut down our generated code a bit. - if matches!(variant.fields, Fields::Unit) { - return None; - } - - let variant_struct_ident = variant_struct_ident(&variant.ident); - let field_types = variant.fields.iter().map(|f| &f.ty); - - let variant_struct = parse_quote! { - #[repr(C)] - struct #variant_struct_ident #impl_generics ( - #core::mem::MaybeUninit<___ZerocopyInnerTag>, - #(#field_types,)* - #phantom_ty, - ) #where_clause; - }; - - // We do this rather than emitting `#[derive(::zerocopy::TryFromBytes)]` - // because that is not hygienic, and this is also more performant. - let try_from_bytes_impl = - derive_try_from_bytes(&ctx.with_input(&variant_struct), Trait::TryFromBytes) - .expect("derive_try_from_bytes should not fail on synthesized type"); - - Some(quote! { - #variant_struct - #try_from_bytes_impl - }) + let candidate = candidate_ident(ctx); + let field_names = fields.iter().enumerate().map(|(idx, field)| { + field + .ident + .as_ref() + .map(|name| name.to_token_stream()) + .unwrap_or_else(|| syn::Index::from(idx).to_token_stream()) }); - - quote! { - #(#variant_structs)* + let field_checks = field_names + .map(|name| { + quote! { + #zerocopy_crate::into_inner!( + #candidate.project::< + #zerocopy_crate::project_clients::TryFromBytesDerive, + _, + { #variant_id }, + { #zerocopy_crate::ident_id!(#name) }, + >() + ).try_into_safe::<_, #zerocopy_crate::BecauseImmutable>() + } + }) + .collect::>(); + // Without invariants, no expression needs the validated pointers. Avoid + // introducing bindings that could collide with caller-defined items. + if ctx.invariant_span.is_none() { + return Ok(quote! { true #(&& #field_checks.is_ok())* }); } -} - -fn variants_union_field_ident(ident: &Ident) -> Ident { - // Field names are prefixed with `__field_` to prevent name collision - // with the `__nonempty` field. - ident!(("__field_{}", ident), ident.span()) -} - -fn generate_variants_union(ctx: &Ctx, data: &DataEnum) -> TokenStream { - let generics = &ctx.ast.generics; - let (_, ty_generics, _) = generics.split_for_impl(); - - let fields = data.variants.iter().filter_map(|variant| { - // We don't generate variant structs for unit variants because we only - // need to check the tag. This helps cut down our generated code a bit. - if matches!(variant.fields, Fields::Unit) { - return None; - } - let field_name = variants_union_field_ident(&variant.ident); - let variant_struct_ident = variant_struct_ident(&variant.ident); - - let core = ctx.core_path(); - Some(quote! { - #field_name: #core::mem::ManuallyDrop<#variant_struct_ident #ty_generics>, + let field_bindings = fields + .iter() + .enumerate() + .map(|(idx, _)| ctx.fresh_ident(&format!("field_{}", idx))) + .collect::>(); + let bound_fields = + fields.iter().copied().zip(field_bindings.iter().cloned()).collect::>(); + let field_validations = fields + .iter() + .enumerate() + .map(|(idx, field)| { + let invariants = crate::invariant::parse(&field.attrs)?; + let invariants = invariants + .iter() + .map(|expression| crate::invariant::bind_fields(&bound_fields[..=idx], expression)); + Ok(quote! { + #( + // Keep `return` in an invariant from bypassing later fields. + if !{ #[inline(always)] || -> #core::primitive::bool { #invariants } }() { + return false; + } + )* + }) }) - }); - - let variants_union = parse_quote! { - #[repr(C)] - union ___ZerocopyVariants #generics { - #(#fields)* - // Enums can have variants with no fields, but unions must - // have at least one field. So we just add a trailing unit - // to ensure that this union always has at least one field. - // Because this union is `repr(C)`, this unit type does not - // affect the layout. - __nonempty: (), + .collect::, Error>>()?; + Ok(quote! { + { + #( + // Retain each shared pointer for subsequent field invariants. + #[allow(unused_variables)] + let #field_bindings = match #field_checks { + #core::result::Result::Ok(#field_bindings) => #field_bindings, + #core::result::Result::Err(_) => return false, + }; + #field_validations + )* + true } - }; - - let has_field = - derive_has_field_struct_union(&ctx.with_input(&variants_union), &variants_union.data); - - quote! { - #variants_union - #has_field - } + }) } -/// Generates an implementation of `is_bit_valid` for an arbitrary enum. -/// -/// The general process is: -/// -/// 1. Generate a tag enum. This is an enum with the same repr, variants, and -/// corresponding discriminants as the original enum, but without any fields -/// on the variants. This gives us access to an enum where the variants have -/// the same discriminants as the one we're writing `is_bit_valid` for. -/// 2. Make constants from the variants of the tag enum. We need these because -/// we can't put const exprs in match arms. -/// 3. Generate variant structs. These are structs which have the same fields as -/// each variant of the enum, and are `#[repr(C)]` with an optional "inner -/// tag". -/// 4. Generate a variants union, with one field for each variant struct type. -/// 5. And finally, our raw enum is a `#[repr(C)]` struct of an "outer tag" and -/// the variants union. +/// Generates an implementation of `is_safe` for an arbitrary enum. /// -/// See these reference links for fully-worked example decompositions. -/// -/// - `repr(C)`: -/// - `repr(int)`: -/// - `repr(C, int)`: -pub(crate) fn derive_is_bit_valid( +/// For an enum with fields, [`derive_enum`] generates the representation model +/// and projection impls. This function reads the tag, matches it against the +/// enum's discriminants, and validates each field of the selected variant +/// through those projections. A fieldless enum needs only the generated tag +/// enum and discriminant constants. +pub(crate) fn derive_is_safe( ctx: &Ctx, data: &DataEnum, repr: &EnumRepr, ) -> Result { - let trait_path = Trait::TryFromBytes.crate_path(ctx); - let tag_enum = generate_tag_enum(ctx, repr, data); - let tag_consts = generate_tag_consts(data); - - let (outer_tag_type, inner_tag_type) = if repr.is_c() { - (quote! { ___ZerocopyTag }, quote! { () }) - } else if repr.is_primitive() { - (quote! { () }, quote! { ___ZerocopyTag }) - } else { + if !(repr.is_c() || repr.is_primitive()) { return Err(Error::new( ctx.ast.span(), "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout", )); - }; - - let variant_structs = generate_variant_structs(ctx, data); - let variants_union = generate_variants_union(ctx, data); - - let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl(); + } let zerocopy_crate = &ctx.zerocopy_crate; - let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None) - .inner_extras(quote! { - type Tag = ___ZerocopyTag; - type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized; - }) - .build(); - let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| { - let variant_ident = &variant.unwrap().ident; - let variants_union_field_ident = variants_union_field_ident(variant_ident); - let field: Box = parse_quote!(()); - fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| { - // Rust does not presently support explicit visibility modifiers on - // enum fields, but we guard against the possibility to ensure this - // derive remains sound. - assert!(matches!(vis, syn::Visibility::Inherited)); - let variant_struct_field_index = Index::from(idx + 1); - let (_, ty_generics, _) = ctx.ast.generics.split_for_impl(); - let has_field_trait = Trait::HasField { - variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }), - // Since Rust does not presently support explicit visibility - // modifiers on enum fields, any public type is suitable here; - // we use `()`. - field: field.clone(), - field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }), - }; - let has_field_path = has_field_trait.crate_path(ctx); - let has_field = ImplBlockBuilder::new( - ctx, - data, - has_field_trait, - FieldBounds::None, - ) - .inner_extras(quote! { - type Type = #ty; + let core = ctx.core_path(); + let candidate = candidate_ident(ctx); + let alignment = ctx.fresh_ident("___ZcAlignment"); + let tag = Ident::new("tag", Span::mixed_site()); + let projections = if data.fields().is_empty() { + let tag_enum = generate_tag_enum(ctx, repr, data); + let tag_consts = generate_tag_consts(data); + quote! { + #tag_enum - #[inline(always)] - fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut ::Type { - use #zerocopy_crate::pointer::cast::{CastSized, Projection}; + type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag< + { #core::mem::size_of::<___ZerocopyTag>() }, + >; - slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>() - .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>() - .project::<_, Projection<_, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field_ident) }>>() - .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>() - .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>() - .as_ptr() - } - }) - .build(); + #tag_consts + } + } else { + derive_enum(ctx, data, Client::TryFromBytesDerive)? + }; - let project = ImplBlockBuilder::new( - ctx, - data, - Trait::ProjectField { - variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }), - // Since Rust does not presently support explicit visibility - // modifiers on enum fields, any public type is suitable - // here; we use `()`. - field: field.clone(), - field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }), - invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)), - }, - FieldBounds::None, - ) - .param_extras(vec![ - parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing), - parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment), - ]) - .inner_extras(quote! { - type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible; - type Invariants = (Aliasing, Alignment, #zerocopy_crate::invariant::Initialized); + let match_arms = data + .variants + .iter() + .enumerate() + .map(|(idx, variant)| { + let name = &variant.ident; + let variant_id = quote! { #zerocopy_crate::ident_id!(#name) }; + let fields_is_safe = derive_variant_is_safe(ctx, &variant_id, &variant.fields)?; + let pattern = if ctx.invariant_span.is_some() { + quote! { #core::option::Option::Some(#idx) } + } else { + tag_ident(name).to_token_stream() + }; + Ok(quote! { + #pattern => { #fields_is_safe } }) - .build(); - - quote! { - #has_field - #project - } }) - }); + .collect::, Error>>()?; - let core = ctx.core_path(); - let match_arms = data.variants.iter().map(|variant| { - let tag_ident = tag_ident(&variant.ident); - let variant_struct_ident = variant_struct_ident(&variant.ident); - let variants_union_field_ident = variants_union_field_ident(&variant.ident); - - if matches!(variant.fields, Fields::Unit) { - // Unit variants don't need any further validation beyond checking - // the tag. - quote! { - #tag_ident => true - } - } else { - quote! { - #tag_ident => { - // SAFETY: Since we know that the tag is `#tag_ident`, we - // know that no other `&`s exist which refer to this enum - // as any other variant. - let variant_md = variants.cast::< - _, - #zerocopy_crate::pointer::cast::Projection< - // #zerocopy_crate::ReadOnly<_>, - _, - { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, - { #zerocopy_crate::ident_id!(#variants_union_field_ident) } - >, - _ - >(); - let variant = variant_md.cast::< - #zerocopy_crate::ReadOnly<#variant_struct_ident #ty_generics>, - #zerocopy_crate::pointer::cast::CastSized, - (#zerocopy_crate::pointer::BecauseRead, _) - >(); - < - #variant_struct_ident #ty_generics as #trait_path - >::is_bit_valid(variant) + let read_tag = quote! { + let #tag = #candidate + .reborrow() + .cast::< + ___ZerocopyTagPrimitive, + #zerocopy_crate::pointer::cast::CastSized, + (#zerocopy_crate::pointer::BecauseRead, _), + >() + .recall_validity::<_, (_, (_, _))>() + .read::<#zerocopy_crate::BecauseImmutable>(); + }; + let tag_init = if ctx.invariant_span.is_some() { + let allow = crate::util::allow_generated_code(); + let tag_arms = data.variants.iter().enumerate().map(|(idx, variant)| { + let tag = tag_ident(&variant.ident); + quote! { #tag => #core::option::Option::Some(#idx) } + }); + quote! { + // Keep implementation-only items and their lint allowances out + // of the scopes containing caller-authored invariant expressions. + #allow + let #tag = { + #projections + #read_tag + match #tag { + #(#tag_arms,)* + _ => #core::option::Option::None, } - } - } - }); - - let generics = &ctx.ast.generics; - let raw_enum: DeriveInput = parse_quote! { - #[repr(C)] - struct ___ZerocopyRawEnum #generics { - tag: ___ZerocopyOuterTag, - variants: ___ZerocopyVariants #ty_generics, + }; } - }; - - let self_ident = &ctx.ast.ident; - let invariants_eq_impl = quote! { - // SAFETY: `___ZerocopyRawEnum` is designed to have the same layout, - // validity, and invariants as `Self`. - unsafe impl #impl_generics #zerocopy_crate::pointer::InvariantsEq<___ZerocopyRawEnum #ty_generics> for #self_ident #ty_generics #where_clause {} - }; - - let raw_enum_projections = - derive_has_field_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data); - - let raw_enum = quote! { - #raw_enum - #invariants_eq_impl - #raw_enum_projections + } else { + quote! { #projections #read_tag } }; Ok(quote! { - // SAFETY: We use `is_bit_valid` to validate that the bit pattern of the + // SAFETY: We use `is_safe` to validate that the bit pattern of the // enum's tag corresponds to one of the enum's discriminants. Then, we // check the bit validity of each field of the corresponding variant. - // Thus, this is a sound implementation of `is_bit_valid`. + // Thus, this is a sound implementation of `is_safe`. #[inline] - fn is_bit_valid<___ZcAlignment>( - mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>, + fn is_safe<#alignment>( + mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>, ) -> #core::primitive::bool where - ___ZcAlignment: #zerocopy_crate::invariant::Alignment, + #alignment: #zerocopy_crate::invariant::Alignment, { - #tag_enum - - type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag< - { #core::mem::size_of::<___ZerocopyTag>() }, - >; - - #tag_consts - - type ___ZerocopyOuterTag = #outer_tag_type; - type ___ZerocopyInnerTag = #inner_tag_type; + #tag_init - #variant_structs - - #variants_union - - #raw_enum - - #has_tag - - #(#has_fields)* - - let tag = { - // SAFETY: - // - The provided cast addresses a subset of the bytes addressed - // by `candidate` because it addresses the starting tag of the - // enum. - // - Because the pointer is cast from `candidate`, it has the - // same provenance as it. - // - There are no `UnsafeCell`s in the tag because it is a - // primitive integer. - // - `tag_ptr` is casted from `candidate`, whose referent is - // `Initialized`. Since we have not written uninitialized - // bytes into the referent, `tag_ptr` is also `Initialized`. - // - // FIXME(#2874): Revise this to a `cast` once `candidate` - // references a `ReadOnly`. - let tag_ptr = unsafe { - candidate.reborrow().project_transmute_unchecked::< - _, - #zerocopy_crate::invariant::Initialized, - #zerocopy_crate::pointer::cast::CastSized - >() - }; - tag_ptr.recall_validity::<_, (_, (_, _))>().read::<#zerocopy_crate::BecauseImmutable>() - }; - - let mut raw_enum = candidate.cast::< - #zerocopy_crate::ReadOnly<___ZerocopyRawEnum #ty_generics>, - #zerocopy_crate::pointer::cast::CastSized, - (#zerocopy_crate::pointer::BecauseRead, _) - >(); - - let variants = #zerocopy_crate::into_inner!(raw_enum.project::< - _, - { #zerocopy_crate::STRUCT_VARIANT_ID }, - { #zerocopy_crate::ident_id!(variants) } - >()); - - match tag { + match #tag { #(#match_arms,)* _ => false, } @@ -447,219 +229,88 @@ pub(crate) fn derive_try_from_bytes(ctx: &Ctx, top_level: Trait) -> Result derive_try_from_bytes_struct(ctx, strct, top_level), Data::Enum(enm) => derive_try_from_bytes_enum(ctx, enm, top_level), - Data::Union(unn) => Ok(derive_try_from_bytes_union(ctx, unn, top_level)), - } -} -fn derive_has_field_struct_union(ctx: &Ctx, data: &dyn DataExt) -> TokenStream { - let fields = ctx.ast.data.fields(); - if fields.is_empty() { - return quote! {}; + Data::Union(unn) => derive_try_from_bytes_union(ctx, unn, top_level), } - - let field_tokens = fields.iter().map(|(vis, ident, _)| { - let ident = ident!(("__z{}", ident), ident.span()); - quote!( - #vis enum #ident {} - ) - }); - - let zerocopy_crate = &ctx.zerocopy_crate; - let variant_id: Box = match &ctx.ast.data { - Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }), - Data::Union(_) => { - let is_repr_c = StructUnionRepr::from_attrs(&ctx.ast.attrs) - .map(|repr| repr.is_c()) - .unwrap_or(false); - if is_repr_c { - parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID }) - } else { - parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID }) - } - } - _ => unreachable!(), - }; - - let core = ctx.core_path(); - let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None) - .inner_extras(quote! { - type Tag = (); - type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit; - }) - .build(); - let has_fields = fields.iter().map(move |(_, ident, ty)| { - let field_token = ident!(("__z{}", ident), ident.span()); - let field: Box = parse_quote!(#field_token); - let field_id: Box = parse_quote!({ #zerocopy_crate::ident_id!(#ident) }); - let has_field_trait = Trait::HasField { - variant_id: variant_id.clone(), - field: field.clone(), - field_id: field_id.clone(), - }; - let has_field_path = has_field_trait.crate_path(ctx); - ImplBlockBuilder::new( - ctx, - data, - has_field_trait, - FieldBounds::None, - ) - .inner_extras(quote! { - type Type = #ty; - - #[inline(always)] - fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut ::Type { - let slf = slf.as_ptr(); - // SAFETY: By invariant on `PtrInner`, `slf` is a non-null - // pointer whose referent is zero-sized or lives in a valid - // allocation. Since `#ident` is a struct or union field of - // `Self`, this projection preserves or shrinks the referent - // size, and so the resulting referent also fits in the same - // allocation. - unsafe { #core::ptr::addr_of_mut!((*slf).#ident) } - } - }).outer_extras(if matches!(&ctx.ast.data, Data::Struct(..)) { - let fields_preserve_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs) - .map(|repr| repr.get_packed().is_none()) - .unwrap(); - let alignment = if fields_preserve_alignment { - quote! { Alignment } - } else { - quote! { #zerocopy_crate::invariant::Unaligned } - }; - // SAFETY: See comments on items. - ImplBlockBuilder::new( - ctx, - data, - Trait::ProjectField { - variant_id: variant_id.clone(), - field, - field_id, - invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)), - }, - FieldBounds::None, - ) - .param_extras(vec![ - parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing), - parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment), - ]) - .inner_extras(quote! { - // SAFETY: Projection into structs is always infallible. - type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible; - // SAFETY: The alignment of the projected `Ptr` is `Unaligned` - // if the structure is packed; otherwise inherited from the - // outer `Ptr`. If the validity of the outer pointer is - // `Initialized`, so too is the validity of its fields. - type Invariants = (Aliasing, #alignment, #zerocopy_crate::invariant::Initialized); - }) - .build() - } else { - quote! {} - }) - .build() - }); - - const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some)) } fn derive_try_from_bytes_struct( ctx: &Ctx, strct: &DataStruct, top_level: Trait, ) -> Result { - let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| { + let extras = if let Some(extras) = try_gen_trivial_is_safe(ctx, top_level) { + extras + } else { let zerocopy_crate = &ctx.zerocopy_crate; - let fields = strct.fields(); - let field_names = fields.iter().map(|(_vis, name, _ty)| name); - let field_tys = fields.iter().map(|(_vis, _name, ty)| ty); + let variant_id = quote! { #zerocopy_crate::STRUCT_VARIANT_ID }; + let fields_is_safe = derive_variant_is_safe(ctx, &variant_id, &strct.fields)?; let core = ctx.core_path(); + let candidate = candidate_ident(ctx); + let alignment = ctx.fresh_ident("___ZcAlignment"); quote!( - // SAFETY: We use `is_bit_valid` to validate that each field is - // bit-valid, and only return `true` if all of them are. The bit - // validity of a struct is just the composition of the bit - // validities of its fields, so this is a sound implementation - // of `is_bit_valid`. + // SAFETY: We use `is_safe` to validate that each field is bit-valid, + // and only return `true` if all of them are. The bit validity of a + // struct is just the composition of the bit validities of its + // fields, so this is a sound implementation of `is_safe`. #[inline] - fn is_bit_valid<___ZcAlignment>( - mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>, + fn is_safe<#alignment>( + mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>, ) -> #core::primitive::bool where - ___ZcAlignment: #zerocopy_crate::invariant::Alignment, + #alignment: #zerocopy_crate::invariant::Alignment, { - true #(&& { - let field_candidate = #zerocopy_crate::into_inner!(candidate.reborrow().project::< - _, - { #zerocopy_crate::STRUCT_VARIANT_ID }, - { #zerocopy_crate::ident_id!(#field_names) } - >()); - <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate) - })* + #fields_is_safe } ) - }); + }; Ok(ImplBlockBuilder::new(ctx, strct, Trait::TryFromBytes, FieldBounds::ALL_SELF) .inner_extras(extras) - .outer_extras(derive_has_field_struct_union(ctx, strct)) + .outer_extras(derive_projection_struct_union(ctx, strct, Client::TryFromBytesDerive)) .build()) } -fn derive_try_from_bytes_union(ctx: &Ctx, unn: &DataUnion, top_level: Trait) -> TokenStream { +fn derive_try_from_bytes_union( + ctx: &Ctx, + unn: &DataUnion, + top_level: Trait, +) -> Result { let field_type_trait_bounds = FieldBounds::All(&[TraitBound::Slf]); let zerocopy_crate = &ctx.zerocopy_crate; - let variant_id: Box = { - let is_repr_c = - StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap_or(false); - if is_repr_c { - parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID }) - } else { - parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID }) - } - }; - - let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| { - let fields = unn.fields(); - let field_names = fields.iter().map(|(_vis, name, _ty)| name); - let field_tys = fields.iter().map(|(_vis, _name, ty)| ty); + let union_variant_id = struct_union_variant_id(ctx).to_token_stream(); + let extras = if let Some(extras) = try_gen_trivial_is_safe(ctx, top_level) { + extras + } else { + let fields_is_safe = unn + .fields + .named + .iter() + .map(|field| derive_variant_is_safe(ctx, &union_variant_id, [field])) + .collect::, _>>()?; let core = ctx.core_path(); + let candidate = candidate_ident(ctx); + let alignment = ctx.fresh_ident("___ZcAlignment"); quote!( - // SAFETY: We use `is_bit_valid` to validate that any field is - // bit-valid; we only return `true` if at least one of them is. - // The bit validity of a union is not yet well defined in Rust, - // but it is guaranteed to be no more strict than this - // definition. See #696 for a more in-depth discussion. + // SAFETY: We use `is_safe` to validate that any field is bit-valid; + // we only return `true` if at least one of them is and its + // invariants also hold. The bit validity of a union is not yet + // well defined in Rust, but it is guaranteed to be no more strict + // than this definition. See #696 for a more in-depth discussion. #[inline] - fn is_bit_valid<___ZcAlignment>( - mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>, + fn is_safe<#alignment>( + mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>, ) -> #core::primitive::bool where - ___ZcAlignment: #zerocopy_crate::invariant::Alignment, + #alignment: #zerocopy_crate::invariant::Alignment, { - false #(|| { - // SAFETY: - // - Since `ReadOnly: Immutable` unconditionally, - // neither `*slf` nor the returned pointer's referent - // permit interior mutation. - // - Both source and destination validity are - // `Initialized`, which is always a sound - // transmutation. - let field_candidate = unsafe { - candidate.reborrow().project_transmute_unchecked::< - _, - _, - #zerocopy_crate::pointer::cast::Projection< - _, - #variant_id, - { #zerocopy_crate::ident_id!(#field_names) } - > - >() - }; - - <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate) - })* + // Keep each field's bindings and early returns local to that + // field, so a failure lets validation try the next field. + false #(|| { #[inline(always)] || { #fields_is_safe } }())* } ) - }); - ImplBlockBuilder::new(ctx, unn, Trait::TryFromBytes, field_type_trait_bounds) + }; + Ok(ImplBlockBuilder::new(ctx, unn, Trait::TryFromBytes, field_type_trait_bounds) .inner_extras(extras) - .outer_extras(derive_has_field_struct_union(ctx, unn)) - .build() + .outer_extras(derive_projection_struct_union(ctx, unn, Client::TryFromBytesDerive)) + .build()) } fn derive_try_from_bytes_enum( ctx: &Ctx, @@ -677,16 +328,15 @@ fn derive_try_from_bytes_enum( .map(|size| enm.fields().is_empty() && enm.variants.len() == 1usize << size) .unwrap_or(false); - let trivial_is_bit_valid = try_gen_trivial_is_bit_valid(ctx, top_level); - let extra = match (trivial_is_bit_valid, could_be_from_bytes) { - (Some(is_bit_valid), _) => is_bit_valid, + let trivial_is_safe = try_gen_trivial_is_safe(ctx, top_level); + let extra = match (trivial_is_safe, could_be_from_bytes) { + (Some(is_safe), _) => is_safe, // SAFETY: It would be sound for the enum to implement `FromBytes`, as - // required by `gen_trivial_is_bit_valid_unchecked`. - (None, true) => unsafe { gen_trivial_is_bit_valid_unchecked(ctx) }, - (None, false) => match derive_is_bit_valid(ctx, enm, &repr) { + // required by `gen_trivial_is_safe_unchecked`. + (None, true) => unsafe { gen_trivial_is_safe_unchecked(ctx) }, + (None, false) => match derive_is_safe(ctx, enm, &repr) { Ok(extra) => extra, - Err(_) if ctx.skip_on_error => return Ok(TokenStream::new()), - Err(e) => return Err(e), + Err(e) => return ctx.error_or_skip(e), }, }; @@ -694,15 +344,14 @@ fn derive_try_from_bytes_enum( .inner_extras(extra) .build()) } -fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option { +fn try_gen_trivial_is_safe(ctx: &Ctx, top_level: Trait) -> Option { // If the top-level trait is `FromBytes` and `Self` has no type parameters, // then the `FromBytes` derive will fail compilation if `Self` is not - // actually soundly `FromBytes`, and so we can rely on that for our - // `is_bit_valid` impl. It's plausible that we could make changes - or Rust - // could make changes (such as the "trivial bounds" language feature) - that - // make this no longer true. To hedge against these, we include an explicit - // `Self: FromBytes` check in the generated `is_bit_valid`, which is - // bulletproof. + // actually soundly `FromBytes`, and so we can rely on that for our `is_safe` + // impl. It's plausible that we could make changes - or Rust could make + // changes (such as the "trivial bounds" language feature) - that make this + // no longer true. To hedge against these, we include an explicit `Self: + // FromBytes` check in the generated `is_safe`, which is bulletproof. // // If `ctx.skip_on_error` is true, we can't rely on the `FromBytes` derive // to fail compilation if `Self` is not actually soundly `FromBytes`. @@ -715,7 +364,7 @@ fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option( + fn is_safe<___ZcAlignment>( _candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>, ) -> #core::primitive::bool where @@ -746,14 +395,14 @@ fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option proc_macro2::TokenStream { +unsafe fn gen_trivial_is_safe_unchecked(ctx: &Ctx) -> proc_macro2::TokenStream { let zerocopy_crate = &ctx.zerocopy_crate; let core = ctx.core_path(); quote!( - // SAFETY: The caller of `gen_trivial_is_bit_valid_unchecked` has - // promised that all initialized bit patterns are valid for `Self`. + // SAFETY: The caller of `gen_trivial_is_safe_unchecked` has promised + // that all initialized bit patterns are valid for `Self`. #[inline(always)] - fn is_bit_valid<___ZcAlignment>( + fn is_safe<___ZcAlignment>( _candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>, ) -> #core::primitive::bool where @@ -763,3 +412,43 @@ unsafe fn gen_trivial_is_bit_valid_unchecked(ctx: &Ctx) -> proc_macro2::TokenStr } ) } + +#[cfg(test)] +mod tests { + use syn::{parse_quote, DeriveInput}; + + use super::*; + + fn derive_error(input: DeriveInput) -> String { + let ctx = Ctx::try_from_derive_input(input).unwrap(); + match derive_try_from_bytes(&ctx, Trait::TryFromBytes) { + Ok(_) => panic!("expected derive to reject conflicting representations"), + Err(error) => error.to_string(), + } + } + + #[test] + fn raw_repr_conflict_matches_ordinary_repr() { + let ordinary = derive_error(parse_quote! { + #[repr(C, align(8))] + #[repr(u8)] + enum Packet { + Value(core::num::NonZeroU8), + End, + } + }); + let raw = derive_error(parse_quote! { + #[repr(C, align(8))] + #[r#repr(r#u8)] + enum Packet { + Value(core::num::NonZeroU8), + End, + } + }); + + // Rejecting both spellings prevents generating a validator whose + // synthetic enum layout disagrees with the compiler's actual layout. + assert_eq!(ordinary, "this conflicts with another representation hint"); + assert_eq!(ordinary, raw); + } +} diff --git a/rust/zerocopy-derive/invariant.rs b/rust/zerocopy-derive/invariant.rs new file mode 100644 index 00000000000000..f7a1a617e37899 --- /dev/null +++ b/rust/zerocopy-derive/invariant.rs @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT +// +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use proc_macro2::{Ident, Span, TokenStream, TokenTree}; +use quote::{quote, ToTokens as _}; +use syn::{ + punctuated::Punctuated, spanned::Spanned as _, Attribute, Data, DeriveInput, Error, Expr, + Field, MacroDelimiter, Meta, Token, +}; + +use crate::util::path_is_ident; + +/// Parses field invariants in attribute order. +pub(crate) fn parse(attrs: &[Attribute]) -> Result, Error> { + let mut invariants = Vec::new(); + for attr in attrs.iter().filter(|attr| path_is_ident(attr.path(), "zerocopy")) { + let options = attr.parse_args_with(Punctuated::::parse_terminated)?; + for option in options { + if !path_is_ident(option.path(), "invariant") { + return Err(Error::new_spanned(option, "expected `invariant(...)`")); + } + match option { + Meta::List(list) if matches!(list.delimiter, MacroDelimiter::Paren(_)) => { + invariants.push(list.parse_args::()?) + } + other => return Err(Error::new_spanned(other, "expected `invariant(...)`")), + } + } + } + Ok(invariants) +} + +/// Validates invariant syntax and placement, returning the first invariant's +/// span so derives which cannot honor invariants can reject them. +pub(crate) fn validate(ast: &DeriveInput) -> Result, Error> { + let fields: Vec<&Field> = match &ast.data { + Data::Struct(data) => data.fields.iter().collect(), + Data::Enum(data) => { + for variant in &data.variants { + if let Some(invariant) = parse(&variant.attrs)?.first() { + return Err(Error::new( + invariant.span(), + "invariants are only supported on named fields", + )); + } + } + data.variants.iter().flat_map(|variant| &variant.fields).collect() + } + Data::Union(data) => data.fields.named.iter().collect(), + }; + let mut first = None; + for field in fields { + if let Some(invariant) = parse(&field.attrs)?.first() { + if field.ident.is_none() { + return Err(Error::new( + invariant.span(), + "invariants are only supported on named fields", + )); + } + first.get_or_insert(invariant.span()); + } + } + Ok(first) +} + +/// Visits identifiers, including those inside macro arguments. +pub(crate) fn idents(tokens: TokenStream) -> Vec { + tokens + .into_iter() + .flat_map(|token| match token { + TokenTree::Ident(ident) => vec![ident], + TokenTree::Group(group) => idents(group.stream()), + _ => Vec::new(), + }) + .collect() +} + +/// Binds semantic field names in every syntax context used by the expression. +/// Preserve the expression's tokens so its other names retain their hygiene. +pub(crate) fn bind_fields(fields: &[(&Field, Ident)], expression: &Expr) -> TokenStream { + let mut body = expression.to_token_stream(); + let references = idents(body.clone()); + for (field, binding) in fields { + let name = match &field.ident { + Some(name) => name, + None => continue, + }; + let spelling = name.to_string(); + let spelling = spelling.trim_start_matches("r#"); + // Also support references produced by a macro invoked by the expression. + let mut call_site = name.clone(); + call_site.set_span(Span::call_site()); + for alias in [name.clone(), call_site].into_iter().chain( + references + .iter() + .filter(|ident| ident.to_string().trim_start_matches("r#") == spelling) + .cloned(), + ) { + body = quote! {{ + // Require a binding; reject names Rust would resolve as constant + // patterns. Item aliases could capture helper calls in macros. + #[allow(unused_variables, non_snake_case, clippy::redundant_pattern)] + let #alias @ _ = #binding; + #body + }}; + } + } + body +} diff --git a/rust/zerocopy-derive/lib.rs b/rust/zerocopy-derive/lib.rs index d387de3683674a..c650b59eaa82c0 100644 --- a/rust/zerocopy-derive/lib.rs +++ b/rust/zerocopy-derive/lib.rs @@ -28,6 +28,10 @@ #![allow(clippy::type_complexity)] // Inlining format args isn't supported on our MSRV. #![allow(clippy::uninlined_format_args)] +// `cargo-zerocopy` supplies this cfg for pinned-nightly tests. During UI tests, +// `testutil::UiTestRunner` explicitly supplies it to the host-built proc macro; +// ordinary `RUSTFLAGS` are not sufficient when Cargo receives `--target`. +#![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(proc_macro_def_site))] #![deny( rustdoc::bare_urls, rustdoc::broken_intra_doc_links, @@ -46,6 +50,7 @@ macro_rules! ident { } mod derive; +mod invariant; #[cfg(test)] mod output_tests; mod repr; @@ -79,7 +84,8 @@ use crate::util::*; /// are currently required to live at the crate root, and so the caller must /// specify the name in order to avoid name collisions. macro_rules! derive { - ($trait:ident => $outer:ident => $inner:path) => { + ($(#[$attr:meta])* $trait:ident => $outer:ident => $inner:path) => { + $(#[$attr])* #[proc_macro_derive($trait, attributes(zerocopy))] pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream { let ast = syn::parse_macro_input!(ts as DeriveInput); @@ -88,10 +94,13 @@ macro_rules! derive { Err(e) => return e.into_compile_error().into(), }; let ts = $inner(&ctx, Trait::$trait).into_ts(); - // We wrap in `const_block` as a backstop in case any derive fails - // to wrap its output in `const_block` (and thus fails to annotate) - // with the full set of `#[allow(...)]` attributes). - let ts = const_block([Some(ts)]); + // Apply generated-code lint allowances as a backstop, except + // around caller-authored invariant expressions. + let ts = if matches!(Trait::$trait, Trait::TryFromBytes) { + ctx.const_block([Some(ts)]) + } else { + const_block([Some(ts)]) + }; #[cfg(test)] crate::util::testutil::check_hygiene(ts.clone()); ts.into() @@ -120,6 +129,7 @@ impl IntoTokenStream for Result { derive!(KnownLayout => derive_known_layout => crate::derive::known_layout::derive); derive!(Immutable => derive_immutable => crate::derive::derive_immutable); +derive!(#[doc(hidden)] Project => derive_project => crate::derive::project::derive); derive!(TryFromBytes => derive_try_from_bytes => crate::derive::try_from_bytes::derive_try_from_bytes); derive!(FromZeros => derive_from_zeros => crate::derive::from_bytes::derive_from_zeros); derive!(FromBytes => derive_from_bytes => crate::derive::from_bytes::derive_from_bytes); @@ -129,6 +139,89 @@ derive!(ByteHash => derive_hash => crate::derive::derive_hash); derive!(ByteEq => derive_eq => crate::derive::derive_eq); derive!(SplitAt => derive_split_at => crate::derive::derive_split_at); +/// Generates a struct whose two identically-printed field types resolve to +/// different types. This is used to test that derives preserve type identity +/// across macro hygiene contexts. +#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)] +#[doc(hidden)] +#[proc_macro] +pub fn __test_hygienically_mixed_into_bytes( + _input: proc_macro::TokenStream, +) -> proc_macro::TokenStream { + use proc_macro::{Group, Ident, Span, TokenStream, TokenTree}; + + fn rewrite(input: TokenStream) -> TokenStream { + input + .into_iter() + .map(|token| match token { + TokenTree::Ident(ident) if ident.to_string() == "CallT" => { + TokenTree::Ident(Ident::new("T", Span::call_site())) + } + TokenTree::Ident(ident) if ident.to_string() == "DefT" => { + TokenTree::Ident(Ident::new("T", Span::def_site())) + } + TokenTree::Group(group) => { + let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream())); + rewritten.set_span(group.span()); + TokenTree::Group(rewritten) + } + token => token, + }) + .collect() + } + + rewrite( + "#[derive(zerocopy_renamed::IntoBytes)] \ + #[zerocopy(crate = \"zerocopy_renamed\")] \ + #[repr(C)] \ + struct IntoBytes15(CallT, DefT);" + .parse() + .expect("test input must parse"), + ) +} + +/// Constructs an invariant whose field declaration and reference have different +/// syntax contexts, while preserving all other caller tokens. +#[doc(hidden)] +#[proc_macro] +pub fn __test_hygienically_mixed_invariant( + input: proc_macro::TokenStream, +) -> proc_macro::TokenStream { + // Cross-compilation does not necessarily pass the nightly test cfg to + // host proc macros. Omit the fixture and its test together in that case. + #[cfg(not(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS))] + { + let _ = input; + proc_macro::TokenStream::new() + } + #[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)] + { + use proc_macro::{Group, Ident, Span, TokenStream, TokenTree}; + + fn rewrite(input: TokenStream) -> TokenStream { + input + .into_iter() + .map(|token| match token { + TokenTree::Ident(ident) if ident.to_string() == "DefField" => { + TokenTree::Ident(Ident::new("field", Span::def_site())) + } + TokenTree::Ident(ident) if ident.to_string() == "CallField" => { + TokenTree::Ident(Ident::new("field", Span::call_site())) + } + TokenTree::Group(group) => { + let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream())); + rewritten.set_span(group.span()); + TokenTree::Group(rewritten) + } + token => token, + }) + .collect() + } + + rewrite(input) + } +} + #[cfg_attr(not(zerocopy_unstable_linux), doc(hidden))] #[proc_macro_derive(most_traits, attributes(zerocopy))] pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream { @@ -153,11 +246,18 @@ pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream { for (derive, t) in derives { tokens.extend(derive(&ctx, t)) } + // Invariants prevent `FromBytes` from generating its usual supertrait + // impls, but still permit checked conversions through `TryFromBytes`. + if ctx.invariant_span.is_some() { + tokens.extend(crate::derive::try_from_bytes::derive_try_from_bytes( + &ctx, + Trait::TryFromBytes, + )); + } - // We wrap in `const_block` as a backstop in case any derive fails - // to wrap its output in `const_block` (and thus fails to annotate) - // with the full set of `#[allow(...)]` attributes). - let ts = const_block([Some(tokens)]); + // Apply generated-code lint allowances as a backstop, except around + // caller-authored invariant expressions. + let ts = ctx.const_block([Some(tokens)]); #[cfg(test)] crate::util::testutil::check_hygiene(ts.clone()); ts.into() diff --git a/rust/zerocopy-derive/repr.rs b/rust/zerocopy-derive/repr.rs index 1525e94302d1b2..a1a877c0f47c61 100644 --- a/rust/zerocopy-derive/repr.rs +++ b/rust/zerocopy-derive/repr.rs @@ -20,6 +20,8 @@ use syn::{ MetaList, }; +use crate::util::{path_is_ident, to_ident_str}; + /// The computed representation of a type. /// /// This is the result of processing all `#[repr(...)]` attributes on a type, if @@ -524,11 +526,11 @@ impl RawRepr { let mut reprs = Vec::new(); for attr in attrs { // Ignore documentation attributes. - if attr.path().is_ident("doc") { + if path_is_ident(attr.path(), "doc") { continue; } if let Meta::List(ref meta_list) = attr.meta { - if meta_list.path.is_ident("repr") { + if path_is_ident(&meta_list.path, "repr") { let parsed: Punctuated = match meta_list.parse_args_with(Punctuated::parse_terminated) { Ok(parsed) => parsed, @@ -564,6 +566,8 @@ impl RawRepr { let ident = path.get_ident().ok_or(UnrecognizedReprError)?; // Only returns `Ok` for non-zero power-of-two values. + // `NonZeroU32::is_power_of_two` is unsupported on our MSRV. + #[allow(clippy::needless_nonzero_get)] let parse_nzu64 = |list: &MetaList| { list.parse_args::() .and_then(|int| int.base10_parse::()) @@ -578,7 +582,8 @@ impl RawRepr { }; use RawRepr::*; - Ok(match (ident.to_string().as_str(), list) { + let ident = to_ident_str(ident); + Ok(match (ident.as_str(), list) { ("u8", None) => U8, ("u16", None) => U16, ("u32", None) => U32, @@ -743,7 +748,40 @@ mod tests { use PrimitiveRepr::*; let nz = |n: u32| NonZeroU32::new(n).unwrap(); + // Rust treats raw identifiers and their ordinary spellings as the + // same representation hints. Exercise every spelling recognized by + // `RawRepr::from_meta` so that adding a hint cannot accidentally + // reintroduce raw-identifier-sensitive parsing. + macro_rules! test_raw_repr { + ($ordinary:meta, $raw:meta => $expected:expr) => {{ + let ordinary: Meta = parse_quote!($ordinary); + let raw: Meta = parse_quote!($raw); + assert_eq!(RawRepr::from_meta(&ordinary).ok().unwrap(), $expected); + assert_eq!(RawRepr::from_meta(&raw).ok().unwrap(), $expected); + }}; + } + + test_raw_repr!(transparent, r#transparent => RawRepr::Transparent); + test_raw_repr!(C, r#C => RawRepr::C); + test_raw_repr!(Rust, r#Rust => RawRepr::Rust); + test_raw_repr!(u8, r#u8 => RawRepr::U8); + test_raw_repr!(u16, r#u16 => RawRepr::U16); + test_raw_repr!(u32, r#u32 => RawRepr::U32); + test_raw_repr!(u64, r#u64 => RawRepr::U64); + test_raw_repr!(u128, r#u128 => RawRepr::U128); + test_raw_repr!(usize, r#usize => RawRepr::Usize); + test_raw_repr!(i8, r#i8 => RawRepr::I8); + test_raw_repr!(i16, r#i16 => RawRepr::I16); + test_raw_repr!(i32, r#i32 => RawRepr::I32); + test_raw_repr!(i64, r#i64 => RawRepr::I64); + test_raw_repr!(i128, r#i128 => RawRepr::I128); + test_raw_repr!(isize, r#isize => RawRepr::Isize); + test_raw_repr!(align(2), r#align(2) => RawRepr::Align(nz(2))); + test_raw_repr!(packed, r#packed => RawRepr::Packed); + test_raw_repr!(packed(2), r#packed(2) => RawRepr::PackedN(nz(2))); + test!(#[repr(transparent)] => StructUnionRepr::Transparent(s())); + test!(#[r#repr(r#transparent)] => StructUnionRepr::Transparent(s())); test!(#[repr()] => StructUnionRepr::Compound(Rust.into(), None)); test!(#[repr(packed)] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(1)).into()))); test!(#[repr(packed(2))] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(2)).into()))); @@ -754,6 +792,8 @@ mod tests { test!(#[repr(C, packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into()))); test!(#[repr(C, align(1))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(1)).into()))); test!(#[repr(C, align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into()))); + test!(#[r#repr(r#C, r#align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into()))); + test!(#[r#repr(r#C, r#packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into()))); test!(#[repr(transparent)] => EnumRepr::Transparent(s())); test!(#[repr()] => EnumRepr::Compound(Rust.into(), None)); diff --git a/rust/zerocopy-derive/util.rs b/rust/zerocopy-derive/util.rs index 5c5e9d3bdcb8f9..717e068dd81c41 100644 --- a/rust/zerocopy-derive/util.rs +++ b/rust/zerocopy-derive/util.rs @@ -13,9 +13,9 @@ use std::num::NonZeroU32; use proc_macro2::{Span, TokenStream}; use quote::{quote, quote_spanned, ToTokens}; use syn::{ - parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error, - Expr, ExprLit, Field, GenericParam, Ident, Index, Lit, LitStr, Meta, Path, Type, Variant, - Visibility, WherePredicate, + parse::ParseBuffer, parse_quote, spanned::Spanned as _, token::PathSep, Data, DataEnum, + DataStruct, DataUnion, DeriveInput, Error, Expr, ExprLit, Field, GenericParam, Ident, Index, + Lit, LitStr, Meta, Path, Type, Variant, Visibility, WherePredicate, }; use crate::repr::{CompoundRepr, EnumRepr, PrimitiveRepr, Repr, Spanned}; @@ -30,6 +30,59 @@ pub(crate) struct Ctx { // The span of the last `#[zerocopy(on_error = ...)]` attribute, if any. pub(crate) on_error_span: Option, + + /// The first field invariant on the source type, if any. + pub(crate) invariant_span: Option, +} + +#[derive(Eq, PartialEq)] +enum CratePath { + External, + CrateRelative, + ModuleRelative, +} + +fn validate_crate_path(path: &Path) -> Result { + if path.segments.is_empty() { + return Err(()); + } + + enum ModuleRelative { + Yes, + No, + } + let first = path.segments[0].ident.to_string(); + let (mut prev_segment, path_type) = match first.as_str() { + "Self" => return Err(()), + "crate" => { + if path.leading_colon.is_some() { + return Err(()); + } + (ModuleRelative::No, CratePath::CrateRelative) + } + "self" | "super" => { + if path.leading_colon.is_some() { + return Err(()); + } + (ModuleRelative::Yes, CratePath::ModuleRelative) + } + _ => (ModuleRelative::No, CratePath::External), + }; + + for seg in path.segments.iter().skip(1) { + let ident = seg.ident.to_string(); + match ident.as_str() { + "Self" | "crate" | "self" => return Err(()), + "super" => match prev_segment { + ModuleRelative::Yes => {} + ModuleRelative::No => return Err(()), + }, + _ => { + prev_segment = ModuleRelative::No; + } + } + } + Ok(path_type) } impl Ctx { @@ -42,24 +95,40 @@ impl Ctx { for attr in &ast.attrs { if let Meta::List(ref meta_list) = attr.meta { - if meta_list.path.is_ident("zerocopy") { + if path_is_ident(&meta_list.path, "zerocopy") { attr.parse_nested_meta(|meta| { - if meta.path.is_ident("crate") { - let expr = meta.value().and_then(|value| value.parse()); + if path_is_ident(&meta.path, "crate") { + let expr = meta.value().and_then(ParseBuffer::parse); if let Ok(Expr::Lit(ExprLit { lit: Lit::Str(lit), .. })) = expr { - if let Ok(path_lit) = lit.parse::() { - path = parse_quote!(::#path_lit); - return Ok(()); + if let Ok(mut path_lit) = lit.parse_with(Path::parse_mod_style) { + if let Ok(crate_path) = validate_crate_path(&path_lit) { + // If not expressly relative, absolutize. + if path_lit.leading_colon.is_none() && crate_path == CratePath::External { + path_lit.leading_colon = Some(PathSep::default()); + } + path = path_lit; + return Ok(()); + } + + return Err(Error::new( + lit.span(), + "`crate` attribute requires a valid module path", + )); } + + return Err(Error::new( + lit.span(), + "`crate` attribute requires a path as the value", + )); } return Err(Error::new( - Span::call_site(), + meta.path.span(), "`crate` attribute requires a path as the value", )); } - if meta.path.is_ident("on_error") { + if path_is_ident(&meta.path, "on_error") { on_error_span = Some(meta.path.span()); let value = meta.value()?; let s: LitStr = value.parse()?; @@ -86,7 +155,8 @@ impl Ctx { } } - Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span }) + let invariant_span = crate::invariant::validate(&ast)?; + Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span, invariant_span }) } pub(crate) fn with_input(&self, input: &DeriveInput) -> Self { @@ -95,6 +165,7 @@ impl Ctx { zerocopy_crate: self.zerocopy_crate.clone(), skip_on_error: self.skip_on_error, on_error_span: self.on_error_span, + invariant_span: self.invariant_span, } } @@ -108,15 +179,41 @@ impl Ctx { quote!(#zerocopy_crate::util::macro_util::core_reexport) } + /// Choose an implementation name absent from the caller's tokens. This is + /// best-effort; names introduced by nested macros are not considered. + /// Unlike local variables, type parameters are not hidden by mixed-site + /// hygiene. + pub(crate) fn fresh_ident(&self, name: &str) -> Ident { + let idents = crate::invariant::idents(self.ast.to_token_stream()); + let mut name = name.to_owned(); + while idents.iter().any(|ident| ident.to_string().trim_start_matches("r#") == name) { + name.push('_'); + } + Ident::new(&name, Span::mixed_site()) + } + + /// Caller-authored invariant expressions must inherit the caller's lint + /// policy, rather than the blanket allowances for generated glue. + pub(crate) fn const_block( + &self, + items: impl IntoIterator>, + ) -> TokenStream { + if self.invariant_span.is_none() { + const_block(items) + } else { + let items = items.into_iter().flatten(); + quote! { const _: () = { #(#items)* }; } + } + } + pub(crate) fn cfg_compile_error(&self) -> TokenStream { // By checking both during the compilation of the proc macro *and* in - // the generated code, we ensure that `--cfg - // zerocopy_unstable_linux` need only be passed *either* when + // the generated code, we ensure that each cfg need only be passed when // compiling this crate *or* when compiling the user's crate. The former // is preferable, but in some situations (such as when cross-compiling // using `cargo build --target`), it doesn't get propagated to this // crate's build by default. - if cfg!(zerocopy_unstable_linux) { + let on_error = if cfg!(zerocopy_unstable_linux) { quote!() } else if let Some(span) = self.on_error_span { let core = self.core_path(); @@ -131,7 +228,24 @@ impl Ctx { } } else { quote!() - } + }; + let invariant = if cfg!(zerocopy_unstable_ptr) { + quote!() + } else if let Some(span) = self.invariant_span { + let core = self.core_path(); + let error_message = + "`invariant` is experimental; pass '--cfg zerocopy_unstable_ptr' to enable"; + quote::quote_spanned! {span=> + #[allow(unused_attributes, unexpected_cfgs)] + const _: () = { + #[cfg(not(zerocopy_unstable_ptr))] + #core::compile_error!(#error_message); + }; + } + } else { + quote!() + }; + quote!(#on_error #invariant) } pub(crate) fn error_or_skip(&self, error: E) -> Result { @@ -151,7 +265,7 @@ pub(crate) trait DataExt { /// FIXME: Extracting field names for enums doesn't really make sense. Types /// makes sense because we don't care about where they live - we just care /// about transitive ownership. But for field names, we'd only use them when - /// generating is_bit_valid, which cares about where they live. + /// generating is_safe, which cares about where they live. fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)>; fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)>; @@ -255,6 +369,18 @@ pub(crate) fn to_ident_str(t: &impl ToString) -> String { } } +/// Does `path` consist solely of the identifier `expected`? +/// +/// Unlike [`Path::is_ident`], this treats a raw identifier and its ordinary +/// spelling as the same identifier. Rust applies that same normalization when +/// interpreting attribute names and arguments. +pub(crate) fn path_is_ident(path: &Path, expected: &str) -> bool { + match path.get_ident() { + Some(ident) => to_ident_str(ident) == expected, + None => false, + } +} + /// This enum describes what kind of padding check needs to be generated for the /// associated impl. pub(crate) enum PaddingCheck { @@ -298,22 +424,46 @@ impl PaddingCheck { } } +#[derive(Copy, Clone)] +pub(crate) enum Client { + ProjectDerive, + TryFromBytesDerive, +} + +impl ToTokens for Client { + fn to_tokens(&self, tokens: &mut TokenStream) { + let s = match self { + Client::ProjectDerive => "ProjectDerive", + Client::TryFromBytesDerive => "TryFromBytesDerive", + }; + let ident = Ident::new(s, Span::call_site()); + tokens.extend(quote!(project_clients::#ident)); + } +} + +impl Client { + pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path { + let zerocopy_crate = &ctx.zerocopy_crate; + parse_quote!(#zerocopy_crate::#self) + } +} + +/// Bundles the trait parameters for `HasField` and `ProjectField`. +#[derive(Clone)] +pub(crate) struct FieldProjection { + pub(crate) variant_id: Box, + pub(crate) field: Box, + pub(crate) field_id: Box, +} + #[derive(Clone)] pub(crate) enum Trait { KnownLayout, - HasTag, - HasField { - variant_id: Box, - field: Box, - field_id: Box, - }, - ProjectField { - variant_id: Box, - field: Box, - field_id: Box, - invariants: Box, - }, + HasTag { client: Client }, + HasField { client: Client, projection: FieldProjection }, + ProjectField { client: Client, projection: FieldProjection, invariants: Box }, Immutable, + Project, TryFromBytes, FromZeros, FromBytes, @@ -340,8 +490,9 @@ impl ToTokens for Trait { Trait::HasField { .. } => "HasField", Trait::ProjectField { .. } => "ProjectField", Trait::KnownLayout => "KnownLayout", - Trait::HasTag => "HasTag", + Trait::HasTag { .. } => "HasTag", Trait::Immutable => "Immutable", + Trait::Project => "Project", Trait::TryFromBytes => "TryFromBytes", Trait::FromZeros => "FromZeros", Trait::FromBytes => "FromBytes", @@ -354,15 +505,18 @@ impl ToTokens for Trait { }; let ident = Ident::new(s, Span::call_site()); let arguments: Option = match self { - Trait::HasField { variant_id, field, field_id } => { - Some(parse_quote!(<#field, #variant_id, #field_id>)) + Trait::HasTag { client } => Some(parse_quote!(<#client>)), + Trait::HasField { client, projection } => { + let FieldProjection { variant_id, field, field_id } = projection; + Some(parse_quote!(<#client, #field, #variant_id, #field_id>)) } - Trait::ProjectField { variant_id, field, field_id, invariants } => { - Some(parse_quote!(<#field, #invariants, #variant_id, #field_id>)) + Trait::ProjectField { client, projection, invariants } => { + let FieldProjection { variant_id, field, field_id } = projection; + Some(parse_quote!(<#client, #field, #invariants, #variant_id, #field_id>)) } Trait::KnownLayout - | Trait::HasTag | Trait::Immutable + | Trait::Project | Trait::TryFromBytes | Trait::FromZeros | Trait::FromBytes @@ -383,6 +537,28 @@ impl Trait { let core = ctx.core_path(); match self { Self::Sized => parse_quote!(#core::marker::#self), + Self::HasTag { client } => { + let client = client.crate_path(ctx); + parse_quote!(#zerocopy_crate::HasTag<#client>) + } + Self::HasField { client, projection } => { + let FieldProjection { variant_id, field, field_id } = projection; + let client = client.crate_path(ctx); + parse_quote!(#zerocopy_crate::HasField<#client, #field, #variant_id, #field_id>) + } + Self::ProjectField { client, projection, invariants } => { + let FieldProjection { variant_id, field, field_id } = projection; + let client = client.crate_path(ctx); + parse_quote!( + #zerocopy_crate::ProjectField< + #client, + #field, + #invariants, + #variant_id, + #field_id + > + ) + } _ => parse_quote!(#zerocopy_crate::#self), } } @@ -704,7 +880,12 @@ impl<'a> ImplBlockBuilder<'a> { let outer_extras = self.outer_extras.filter(|e| !e.is_empty()); let cfg_compile_error = self.ctx.cfg_compile_error(); - const_block([Some(cfg_compile_error), Some(impl_tokens), outer_extras]) + let items = [Some(cfg_compile_error), Some(impl_tokens), outer_extras]; + if matches!(self.trt, Trait::TryFromBytes) { + self.ctx.const_block(items) + } else { + const_block(items) + } } } @@ -730,8 +911,7 @@ impl BoolExt for bool { } } -pub(crate) fn const_block(items: impl IntoIterator>) -> TokenStream { - let items = items.into_iter().flatten(); +pub(crate) fn allow_generated_code() -> TokenStream { quote! { #[allow( // FIXME(#553): Add a test that generates a warning when @@ -748,6 +928,14 @@ pub(crate) fn const_block(items: impl IntoIterator>) non_ascii_idents, clippy::missing_inline_in_public_items, )] + } +} + +pub(crate) fn const_block(items: impl IntoIterator>) -> TokenStream { + let items = items.into_iter().flatten(); + let allow = allow_generated_code(); + quote! { + #allow #[deny(ambiguous_associated_items)] // While there are not currently any warnings that this suppresses // (that we're aware of), it's good future-proofing hygiene. @@ -778,7 +966,8 @@ pub(crate) fn generate_tag_enum(ctx: &Ctx, repr: &EnumRepr, data: &DataEnum) -> quote! { #repr - #[allow(dead_code)] + #[allow(dead_code, clippy::derive_partial_eq_without_eq)] + #[derive(Copy, Clone, PartialEq)] pub enum ___ZerocopyTag { #(#variants,)* } @@ -824,18 +1013,40 @@ pub(crate) mod testutil { pub(crate) fn check_hygiene(ts: TokenStream) { struct AmbiguousItemVisitor; + fn panic_ambiguous_path(path: &impl quote::ToTokens) -> ! { + panic!( + "Found ambiguous path `{}` in generated output. \ + All associated item access must be fully qualified (e.g., `::Item`) \ + to prevent hygiene issues.", + quote::quote!(#path) + ); + } + impl<'ast> Visit<'ast> for AmbiguousItemVisitor { + fn visit_expr_path(&mut self, i: &'ast syn::ExprPath) { + if let Some(qself) = &i.qself { + if qself.position == 0 { + panic_ambiguous_path(i); + } + } + visit::visit_expr_path(self, i); + } + fn visit_path(&mut self, i: &'ast syn::Path) { if i.segments.len() > 1 && i.segments.first().unwrap().ident == "Self" { - panic!( - "Found ambiguous path `{}` in generated output. \ - All associated item access must be fully qualified (e.g., `::Item`) \ - to prevent hygiene issues.", - quote::quote!(#i) - ); + panic_ambiguous_path(i); } visit::visit_path(self, i); } + + fn visit_type_path(&mut self, i: &'ast syn::TypePath) { + if let Some(qself) = &i.qself { + if qself.position == 0 { + panic_ambiguous_path(i); + } + } + visit::visit_type_path(self, i); + } } let file = syn::parse2::(ts).expect("failed to parse generated output as File"); @@ -860,4 +1071,117 @@ pub(crate) mod testutil { } }); } + + #[test] + #[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")] + fn test_check_hygiene_type_relative_expr_failure() { + check_hygiene(quote::quote! { + fn foo() { + let _ = ::Ambiguous; + } + }); + } + + #[test] + #[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")] + fn test_check_hygiene_type_relative_type_failure() { + check_hygiene(quote::quote! { + fn foo() { + let _: ::Ambiguous; + } + }); + } + + #[test] + fn test_validate_crate_path() { + use syn::parse_str; + + let valid = [ + "zerocopy", + "crate", + "crate::foo::bar", + "self", + "self::foo", + "self::super::foo", + "super", + "super::foo", + "super::super::foo", + "super::super::super", + "foo::bar::baz", + "::foo::bar", + ]; + + for path_str in valid { + let path = parse_str::(path_str).unwrap(); + assert!( + super::validate_crate_path(&path).is_ok(), + "expected valid path for `{}`", + path_str + ); + } + + let invalid = [ + "::crate::foo", + "::self::foo", + "::super::foo", + "Self", + "Self::foo", + "foo::Self::bar", + "foo::crate::bar", + "foo::super::bar", + "foo::self", + "super::foo::super", + "super::crate::foo", + "crate::super::foo", + "self::self::foo", + ]; + + for path_str in invalid { + let path = parse_str::(path_str).unwrap(); + assert!( + super::validate_crate_path(&path).is_err(), + "expected invalid path for `{}`", + path_str + ); + } + } + + #[test] + fn test_path_is_ident() { + use syn::parse_str; + + for (expected, ordinary, raw) in [ + ("doc", "doc", "r#doc"), + ("repr", "repr", "r#repr"), + ("zerocopy", "zerocopy", "r#zerocopy"), + ("on_error", "on_error", "r#on_error"), + ] { + let ordinary = parse_str::(ordinary).unwrap(); + let raw = parse_str::(raw).unwrap(); + assert!(super::path_is_ident(&ordinary, expected)); + assert!(super::path_is_ident(&raw, expected)); + } + + // `r#crate` is forbidden by Rust's raw identifier grammar. + let crate_path = parse_str::("crate").unwrap(); + assert!(super::path_is_ident(&crate_path, "crate")); + + for path in ["::zerocopy", "module::zerocopy"] { + let path = parse_str::(path).unwrap(); + assert!(!super::path_is_ident(&path, "zerocopy")); + } + } + + #[test] + fn test_raw_zerocopy_attributes() { + let ast: syn::DeriveInput = syn::parse_quote! { + #[r#zerocopy(crate = "renamed", r#on_error = "skip")] + struct Foo; + }; + + let ctx = super::Ctx::try_from_derive_input(ast).unwrap(); + let path = &ctx.zerocopy_crate; + assert_eq!(quote::quote!(#path).to_string(), ":: renamed"); + assert!(ctx.skip_on_error); + } } diff --git a/rust/zerocopy/README.md b/rust/zerocopy/README.md index 3d11a6502cf0cb..5cdea261e45fec 100644 --- a/rust/zerocopy/README.md +++ b/rust/zerocopy/README.md @@ -1,13 +1,13 @@ # `zerocopy` -These source files come from the Rust `zerocopy` crate, version v0.8.54 -(released 2026-07-08), hosted in the +These source files come from the Rust `zerocopy` crate, version v0.8.62 +(released 2026-10-08), hosted in the repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only modified to tweak the SPDX license identifiers. For copyright details, please see: - https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1 - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE - https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT + https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1 + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE + https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 index f68bad612695c7..12ab2e3c4b3705 100644 --- a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 @@ -1,5 +1,4 @@ bench_as_bytes_dynamic_size: mov rax, rdi - lea rdx, [2*rsi + 5] - and rdx, -2 + lea rdx, [2*rsi + 4] ret diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca index c3b92a9a95fe18..d661f093fea6f8 100644 --- a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 -Instructions: 400 -Total Cycles: 137 -Total uOps: 400 +Instructions: 300 +Total Cycles: 104 +Total uOps: 300 Dispatch Width: 4 -uOps Per Cycle: 2.92 -IPC: 2.92 +uOps Per Cycle: 2.88 +IPC: 2.88 Block RThroughput: 1.0 @@ -19,8 +19,7 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 mov rax, rdi - 1 1 0.50 lea rdx, [2*rsi + 5] - 1 1 0.33 and rdx, -2 + 1 1 0.50 lea rdx, [2*rsi + 4] 1 1 1.00 U ret @@ -37,11 +36,10 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 1.33 1.33 - 1.34 - - + - - 0.99 1.00 - 1.01 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 0.66 - 0.34 - - mov rax, rdi - - - 0.33 0.67 - - - - lea rdx, [2*rsi + 5] - - - 1.00 - - - - - and rdx, -2 + - - 0.99 - - 0.01 - - mov rax, rdi + - - - 1.00 - - - - lea rdx, [2*rsi + 4] - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64 b/rust/zerocopy/benches/extend_vec_zeroed.x86-64 index 831b2a075fec69..0790ce20273ddb 100644 --- a/rust/zerocopy/benches/extend_vec_zeroed.x86-64 +++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64 @@ -41,8 +41,8 @@ bench_extend_vec_zeroed: .LBB6_6: mov r13, rsi lea rcx, [rax + rax] - cmp r15, rcx - cmova rcx, r15 + cmp rcx, r15 + cmovbe rcx, r15 cmp rcx, 5 mov r14d, 4 cmovae r14, rcx diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca index cfab1eea8f56e3..cf3c5a5ad5e368 100644 --- a/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca +++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca @@ -55,8 +55,8 @@ Instruction Info: 1 1 1.00 U ret 1 1 0.33 mov r13, rsi 1 1 0.50 lea rcx, [rax + rax] - 1 1 0.33 cmp r15, rcx - 3 3 1.00 cmova rcx, r15 + 1 1 0.33 cmp rcx, r15 + 3 3 1.00 cmovbe rcx, r15 1 1 0.33 cmp rcx, 5 1 1 0.33 mov r14d, 4 2 2 0.67 cmovae r14, rcx @@ -128,8 +128,8 @@ Resource pressure by instruction: - - - - - 1.00 - - ret - - 1.00 - - - - - mov r13, rsi - - 0.01 0.99 - - - - lea rcx, [rax + rax] - - - 0.99 0.01 - - - - cmp r15, rcx - - - 2.00 0.01 - 0.99 - - cmova rcx, r15 + - - 0.99 0.01 - - - - cmp rcx, r15 + - - 2.00 0.01 - 0.99 - - cmovbe rcx, r15 - - 0.01 0.99 - - - - cmp rcx, 5 - - 0.01 0.99 - - - - mov r14d, 4 - - 1.00 0.01 - 0.99 - - cmovae r14, rcx diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64 b/rust/zerocopy/benches/insert_vec_zeroed.x86-64 index 9db87403cb55b5..b5426c3f4a3e44 100644 --- a/rust/zerocopy/benches/insert_vec_zeroed.x86-64 +++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64 @@ -54,8 +54,8 @@ bench_insert_vec_zeroed: mov r15, rsi mov rbp, rdx lea rcx, [rax + rax] - cmp r12, rcx - cmova rcx, r12 + cmp rcx, r12 + cmovbe rcx, r12 cmp rcx, 5 mov r14d, 4 cmovae r14, rcx diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca index 665240667844b2..a6c17e236823ad 100644 --- a/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca +++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca @@ -68,8 +68,8 @@ Instruction Info: 1 1 0.33 mov r15, rsi 1 1 0.33 mov rbp, rdx 1 1 0.50 lea rcx, [rax + rax] - 1 1 0.33 cmp r12, rcx - 3 3 1.00 cmova rcx, r12 + 1 1 0.33 cmp rcx, r12 + 3 3 1.00 cmovbe rcx, r12 1 1 0.33 cmp rcx, 5 1 1 0.33 mov r14d, 4 2 2 0.67 cmovae r14, rcx @@ -159,8 +159,8 @@ Resource pressure by instruction: - - 0.49 0.51 - - - - mov r15, rsi - - 0.51 0.49 - - - - mov rbp, rdx - - 0.49 0.51 - - - - lea rcx, [rax + rax] - - - 0.49 0.50 - 0.01 - - cmp r12, rcx - - - 1.04 0.50 - 1.46 - - cmova rcx, r12 + - - 0.49 0.50 - 0.01 - - cmp rcx, r12 + - - 1.04 0.50 - 1.46 - - cmovbe rcx, r12 - - 0.49 0.49 - 0.02 - - cmp rcx, 5 - - 0.50 - - 0.50 - - mov r14d, 4 - - 0.50 0.51 - 0.99 - - cmovae r14, rcx diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64 b/rust/zerocopy/benches/new_vec_zeroed.x86-64 index b5c083aa0d36d8..edf421e3192c05 100644 --- a/rust/zerocopy/benches/new_vec_zeroed.x86-64 +++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64 @@ -29,8 +29,7 @@ bench_new_vec_zeroed: mov qword ptr [rax + 16], rsi ret .LBB5_5: - movabs rcx, -9223372036854775808 - mov qword ptr [rax], rcx + mov qword ptr [rax], -1 ret .LBB5_2: mov ecx, 2 diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca index b4fb4544ec39a9..f0e47d004cccdc 100644 --- a/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca +++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 -Instructions: 3700 +Instructions: 3600 Total Cycles: 3486 -Total uOps: 4600 +Total uOps: 4500 Dispatch Width: 4 -uOps Per Cycle: 1.32 -IPC: 1.06 +uOps Per Cycle: 1.29 +IPC: 1.03 Block RThroughput: 12.0 @@ -47,8 +47,7 @@ Instruction Info: 1 1 1.00 * mov qword ptr [rax + 8], rcx 1 1 1.00 * mov qword ptr [rax + 16], rsi 1 1 1.00 U ret - 1 1 0.33 movabs rcx, -9223372036854775808 - 1 1 1.00 * mov qword ptr [rax], rcx + 1 1 1.00 * mov qword ptr [rax], -1 1 1 1.00 U ret 1 1 0.33 mov ecx, 2 1 1 1.00 * mov qword ptr [rax], rsi @@ -70,31 +69,31 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.99 6.99 12.00 10.02 8.00 9.00 + - - 5.99 6.01 12.00 11.00 8.00 9.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.01 0.98 - 0.01 - - mov rax, rdi - - - 0.98 0.02 - - - - movabs rcx, 1537228672809129301 - - - 0.02 0.98 - - - - cmp rsi, rcx + - - 0.97 0.02 - 0.01 - - mov rax, rdi + - - 0.02 0.98 - - - - movabs rcx, 1537228672809129301 + - - 0.97 0.03 - - - - cmp rsi, rcx - - - - - 1.00 - - ja .LBB5_5 - - - 0.98 - - 0.02 - - test rsi, rsi + - - 0.03 0.96 - 0.01 - - test rsi, rsi - - - - - 1.00 - - je .LBB5_2 - - - - 1.00 - - 1.00 push r15 - - - - 1.00 - 1.00 - push r14 - - - - 1.00 - - 1.00 push rbx - - - - 1.00 - - - - lea rcx, [rsi + rsi] - - - - 1.00 - - - - lea rbx, [rcx + 2*rcx] - - - 1.00 - - - - - mov r14, rax - - - 1.00 - - - - - mov r15, rsi + - - 0.99 0.01 - - - - lea rcx, [rsi + rsi] + - - 0.99 0.01 - - - - lea rbx, [rcx + 2*rcx] + - - 0.01 0.99 - - - - mov r14, rax + - - 0.01 0.01 - 0.98 - - mov r15, rsi - - - - 1.00 1.00 2.00 - call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL] - - - - 0.01 - 0.99 - - mov esi, 2 - - - 0.01 0.99 - - - - mov rdi, rbx + - - - 0.98 - 0.02 - - mov esi, 2 + - - 0.96 0.02 - 0.02 - - mov rdi, rbx - - - - 1.00 1.00 - 2.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL] - - - 0.01 - - 0.99 - - mov rsi, r15 - - - 0.99 0.01 - - - - mov rcx, rax - - - - 0.99 - 0.01 - - mov rax, r14 - - - 0.99 0.01 - - - - test rcx, rcx + - - 0.02 0.96 - 0.02 - - mov rsi, r15 + - - 0.02 0.02 - 0.96 - - mov rcx, rax + - - 0.96 0.02 - 0.02 - - mov rax, r14 + - - 0.02 0.02 - 0.96 - - test rcx, rcx - - - - - - - 1.00 pop rbx - - - - - - 1.00 - pop r14 - - - - - - - 1.00 pop r15 @@ -103,10 +102,9 @@ Resource pressure by instruction: - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi - - - - - 1.00 - - ret - - - 0.01 0.99 - - - - movabs rcx, -9223372036854775808 - - - - - 1.00 - - 1.00 mov qword ptr [rax], rcx + - - - - 1.00 - - 1.00 mov qword ptr [rax], -1 - - - - - 1.00 - - ret - - - 0.99 0.01 - - - - mov ecx, 2 + - - 0.02 0.98 - - - - mov ecx, 2 - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 index e844a4608fac3d..7f695f942e67e7 100644 --- a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 @@ -1,22 +1,21 @@ bench_ref_from_bytes_dynamic_padding: - test dil, 3 - jne .LBB5_3 - movabs rax, 9223372036854775804 - and rax, rsi - cmp rax, 9 - jb .LBB5_3 - add rax, -9 + cmp rsi, 12 + setae al + mov ecx, esi + or ecx, edi + test cl, 3 + sete cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor edi, edi + mov rax, rdi + ret +.LBB5_2: + add rsi, -9 movabs rcx, -6148914691236517205 + mov rax, rsi mul rcx shr rdx - lea rax, [rdx + 2*rdx] - or rax, 3 - add rax, 9 - cmp rsi, rax - je .LBB5_4 -.LBB5_3: - xor edi, edi - mov rdx, rsi -.LBB5_4: mov rax, rdi ret diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca index 423ed38ba28d11..5eb4fd1a113d2f 100644 --- a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 Instructions: 1900 -Total Cycles: 645 +Total Cycles: 641 Total uOps: 2000 Dispatch Width: 4 -uOps Per Cycle: 3.10 -IPC: 2.95 +uOps Per Cycle: 3.12 +IPC: 2.96 Block RThroughput: 5.0 @@ -18,23 +18,23 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 test dil, 3 - 1 1 1.00 jne .LBB5_3 - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rax, rsi - 1 1 0.33 cmp rax, 9 - 1 1 1.00 jb .LBB5_3 - 1 1 0.33 add rax, -9 + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setae al + 1 1 0.33 mov ecx, esi + 1 1 0.33 or ecx, edi + 1 1 0.33 test cl, 3 + 1 1 0.50 sete cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor edi, edi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + 1 1 0.33 add rsi, -9 1 1 0.33 movabs rcx, -6148914691236517205 + 1 1 0.33 mov rax, rsi 2 4 1.00 mul rcx 1 1 0.50 shr rdx - 1 1 0.50 lea rax, [rdx + 2*rdx] - 1 1 0.33 or rax, 3 - 1 1 0.33 add rax, 9 - 1 1 0.33 cmp rsi, rax - 1 1 1.00 je .LBB5_4 - 1 0 0.25 xor edi, edi - 1 1 0.33 mov rdx, rsi 1 1 0.33 mov rax, rdi 1 1 1.00 U ret @@ -52,26 +52,26 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.32 6.33 - 6.35 - - + - - 6.33 6.33 - 6.34 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.64 0.35 - 0.01 - - test dil, 3 - - - - - - 1.00 - - jne .LBB5_3 - - - 0.34 0.65 - 0.01 - - movabs rax, 9223372036854775804 - - - 0.35 0.65 - - - - and rax, rsi - - - 0.33 0.34 - 0.33 - - cmp rax, 9 - - - - - - 1.00 - - jb .LBB5_3 - - - 0.35 - - 0.65 - - add rax, -9 - - - 0.97 0.01 - 0.02 - - movabs rcx, -6148914691236517205 - - - 1.00 1.00 - - - - mul rcx - - - 0.99 - - 0.01 - - shr rdx - - - 0.33 0.67 - - - - lea rax, [rdx + 2*rdx] - - - 0.34 0.66 - - - - or rax, 3 - - - 0.33 0.66 - 0.01 - - add rax, 9 - - - 0.01 0.99 - - - - cmp rsi, rax - - - - - - 1.00 - - je .LBB5_4 + - - 0.99 - - 0.01 - - cmp rsi, 12 + - - 1.00 - - - - - setae al + - - - 0.34 - 0.66 - - mov ecx, esi + - - - 0.99 - 0.01 - - or ecx, edi + - - - 1.00 - - - - test cl, 3 + - - 0.67 - - 0.33 - - sete cl + - - - 1.00 - - - - and cl, al + - - - 1.00 - - - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 - - - - - - - - xor edi, edi - - - 0.32 0.01 - 0.67 - - mov rdx, rsi - - - 0.02 0.34 - 0.64 - - mov rax, rdi + - - 0.66 - - 0.34 - - mov rax, rdi + - - - - - 1.00 - - ret + - - 0.35 0.01 - 0.64 - - add rsi, -9 + - - 0.65 0.33 - 0.02 - - movabs rcx, -6148914691236517205 + - - 0.01 0.33 - 0.66 - - mov rax, rsi + - - 1.00 1.00 - - - - mul rcx + - - 1.00 - - - - - shr rdx + - - - 0.33 - 0.67 - - mov rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 index cc905b76c06fbc..43cadf35278396 100644 --- a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 @@ -1,20 +1,11 @@ bench_ref_from_bytes_dynamic_size: mov rdx, rsi - cmp rsi, 4 - setb al - or al, dil - test al, 1 - je .LBB5_2 + mov ecx, edx + or ecx, edi xor eax, eax - ret -.LBB5_2: - lea rcx, [rdx - 4] - mov rsi, rcx - and rsi, -2 - add rsi, 4 - shr rcx - xor eax, eax - cmp rdx, rsi - cmove rdx, rcx + sub rdx, 4 + cmovb rdi, rax + shr rdx + test cl, 1 cmove rax, rdi ret diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca index 68aea583e40123..444f4fad4ff399 100644 --- a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1800 -Total Cycles: 704 -Total uOps: 2000 +Instructions: 1000 +Total Cycles: 373 +Total uOps: 1200 Dispatch Width: 4 -uOps Per Cycle: 2.84 -IPC: 2.56 -Block RThroughput: 5.0 +uOps Per Cycle: 3.22 +IPC: 2.68 +Block RThroughput: 3.0 Instruction Info: @@ -19,21 +19,13 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 mov rdx, rsi - 1 1 0.33 cmp rsi, 4 - 1 1 0.50 setb al - 1 1 0.33 or al, dil - 1 1 0.33 test al, 1 - 1 1 1.00 je .LBB5_2 + 1 1 0.33 mov ecx, edx + 1 1 0.33 or ecx, edi 1 0 0.25 xor eax, eax - 1 1 1.00 U ret - 1 1 0.50 lea rcx, [rdx - 4] - 1 1 0.33 mov rsi, rcx - 1 1 0.33 and rsi, -2 - 1 1 0.33 add rsi, 4 - 1 1 0.50 shr rcx - 1 0 0.25 xor eax, eax - 1 1 0.33 cmp rdx, rsi - 2 2 0.67 cmove rdx, rcx + 1 1 0.33 sub rdx, 4 + 2 2 0.67 cmovb rdi, rax + 1 1 0.50 shr rdx + 1 1 0.33 test cl, 1 2 2 0.67 cmove rax, rdi 1 1 1.00 U ret @@ -51,25 +43,17 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 5.97 5.98 - 6.05 - - + - - 3.66 3.67 - 3.67 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.97 0.01 - 0.02 - - mov rdx, rsi - - - 0.01 0.02 - 0.97 - - cmp rsi, 4 - - - 0.03 - - 0.97 - - setb al - - - 0.01 0.02 - 0.97 - - or al, dil - - - - 0.98 - 0.02 - - test al, 1 - - - - - - 1.00 - - je .LBB5_2 - - - - - - - - - xor eax, eax - - - - - - 1.00 - - ret - - - 0.98 0.02 - - - - lea rcx, [rdx - 4] - - - 0.01 0.99 - - - - mov rsi, rcx - - - - 0.98 - 0.02 - - and rsi, -2 - - - 0.98 0.01 - 0.01 - - add rsi, 4 - - - 0.99 - - 0.01 - - shr rcx + - - 0.32 0.01 - 0.67 - - mov rdx, rsi + - - 0.33 0.35 - 0.32 - - mov ecx, edx + - - 0.01 0.34 - 0.65 - - or ecx, edi - - - - - - - - xor eax, eax - - - 0.02 0.97 - 0.01 - - cmp rdx, rsi - - - 0.99 0.99 - 0.02 - - cmove rdx, rcx - - - 0.98 0.99 - 0.03 - - cmove rax, rdi + - - 0.67 0.33 - - - - sub rdx, 4 + - - 0.99 0.98 - 0.03 - - cmovb rdi, rax + - - 0.03 - - 0.97 - - shr rdx + - - 0.32 0.67 - 0.01 - - test cl, 1 + - - 0.99 0.99 - 0.02 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 index 2c8da68c8b5b89..0e16b103344f79 100644 --- a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 +++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 @@ -1,8 +1,7 @@ bench_ref_from_bytes_static_size: - mov ecx, edi - and ecx, 1 - xor rsi, 6 xor eax, eax - or rsi, rcx + test dil, 1 + cmovne rdi, rax + cmp rsi, 6 cmove rax, rdi ret diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca index 832697801ee209..f1e118abf09a86 100644 --- a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 -Instructions: 700 -Total Cycles: 240 +Instructions: 600 +Total Cycles: 305 Total uOps: 800 Dispatch Width: 4 -uOps Per Cycle: 3.33 -IPC: 2.92 +uOps Per Cycle: 2.62 +IPC: 1.97 Block RThroughput: 2.0 @@ -18,11 +18,10 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 mov ecx, edi - 1 1 0.33 and ecx, 1 - 1 1 0.33 xor rsi, 6 1 0 0.25 xor eax, eax - 1 1 0.33 or rsi, rcx + 1 1 0.33 test dil, 1 + 2 2 0.67 cmovne rdi, rax + 1 1 0.33 cmp rsi, 6 2 2 0.67 cmove rax, rdi 1 1 1.00 U ret @@ -40,14 +39,13 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 2.33 2.33 - 2.34 - - + - - 1.95 2.28 - 2.77 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.01 0.98 - 0.01 - - mov ecx, edi - - - 0.02 0.66 - 0.32 - - and ecx, 1 - - - 0.33 0.66 - 0.01 - - xor rsi, 6 - - - - - - - - xor eax, eax - - - 0.98 0.02 - - - - or rsi, rcx - - - 0.99 0.01 - 1.00 - - cmove rax, rdi + - - 0.05 0.06 - 0.89 - - test dil, 1 + - - 0.95 0.94 - 0.11 - - cmovne rdi, rax + - - - 0.34 - 0.66 - - cmp rsi, 6 + - - 0.95 0.94 - 0.11 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 index a58592a245032e..e7f7820a69d125 100644 --- a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 @@ -1,19 +1,19 @@ bench_ref_from_prefix_dynamic_padding: + test dil, 3 + sete al + cmp rsi, 12 + setae cl + test al, cl + jne .LBB5_2 xor edx, edx - mov eax, 0 test dil, 3 - je .LBB5_1 - ret -.LBB5_1: - movabs rax, 9223372036854775804 - and rsi, rax - cmp rsi, 9 - jae .LBB5_3 - mov edx, 1 - xor eax, eax + sete dl + xor edi, edi + mov rax, rdi ret -.LBB5_3: - add rsi, -9 +.LBB5_2: + or rsi, 3 + add rsi, -12 movabs rcx, -6148914691236517205 mov rax, rsi mul rcx diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca index 62ea4babaf289c..a3d68f37a8be40 100644 --- a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1900 -Total Cycles: 608 -Total uOps: 2000 +Instructions: 2000 +Total Cycles: 640 +Total uOps: 2100 Dispatch Width: 4 -uOps Per Cycle: 3.29 +uOps Per Cycle: 3.28 IPC: 3.13 -Block RThroughput: 5.0 +Block RThroughput: 5.3 Instruction Info: @@ -18,19 +18,20 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 test dil, 3 + 1 1 0.50 sete al + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setae cl + 1 1 0.33 test al, cl + 1 1 1.00 jne .LBB5_2 1 0 0.25 xor edx, edx - 1 1 0.33 mov eax, 0 1 1 0.33 test dil, 3 - 1 1 1.00 je .LBB5_1 - 1 1 1.00 U ret - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rsi, rax - 1 1 0.33 cmp rsi, 9 - 1 1 1.00 jae .LBB5_3 - 1 1 0.33 mov edx, 1 - 1 0 0.25 xor eax, eax + 1 1 0.50 sete dl + 1 0 0.25 xor edi, edi + 1 1 0.33 mov rax, rdi 1 1 1.00 U ret - 1 1 0.33 add rsi, -9 + 1 1 0.33 or rsi, 3 + 1 1 0.33 add rsi, -12 1 1 0.33 movabs rcx, -6148914691236517205 1 1 0.33 mov rax, rsi 2 4 1.00 mul rcx @@ -52,26 +53,27 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.00 6.00 - 6.00 - - + - - 6.33 6.33 - 6.34 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.65 0.34 - 0.01 - - test dil, 3 + - - 0.99 - - 0.01 - - sete al + - - 0.01 0.33 - 0.66 - - cmp rsi, 12 + - - 0.34 - - 0.66 - - setae cl + - - 1.00 - - - - - test al, cl + - - - - - 1.00 - - jne .LBB5_2 - - - - - - - - xor edx, edx - - - 0.01 0.98 - 0.01 - - mov eax, 0 - - - 0.98 0.01 - 0.01 - - test dil, 3 - - - - - - 1.00 - - je .LBB5_1 - - - - - - 1.00 - - ret - - - 0.01 0.99 - - - - movabs rax, 9223372036854775804 - - - - 1.00 - - - - and rsi, rax - - - - 1.00 - - - - cmp rsi, 9 - - - - - - 1.00 - - jae .LBB5_3 - - - 1.00 - - - - - mov edx, 1 - - - - - - - - - xor eax, eax + - - - 0.99 - 0.01 - - test dil, 3 + - - - - - 1.00 - - sete dl + - - - - - - - - xor edi, edi + - - - 1.00 - - - - mov rax, rdi - - - - - 1.00 - - ret - - - 0.02 0.02 - 0.96 - - add rsi, -9 - - - 0.99 0.01 - - - - movabs rcx, -6148914691236517205 - - - 0.01 0.99 - - - - mov rax, rsi + - - 0.01 0.99 - - - - or rsi, 3 + - - 0.33 0.67 - - - - add rsi, -12 + - - 0.66 0.34 - - - - movabs rcx, -6148914691236517205 + - - 0.34 0.33 - 0.33 - - mov rax, rsi - - 1.00 1.00 - - - - mul rcx - - 1.00 - - - - - shr rdx - - - 0.98 - - 0.02 - - mov rax, rdi + - - - 0.34 - 0.66 - - mov rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 index fe6332c9100ce2..3f2e8746789746 100644 --- a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 @@ -1,17 +1,10 @@ bench_ref_from_prefix_dynamic_size: - xor edx, edx - mov eax, 0 - test dil, 1 - jne .LBB5_4 - cmp rsi, 4 - jae .LBB5_3 - mov edx, 1 - xor eax, eax - ret -.LBB5_3: - add rsi, -4 - shr rsi mov rdx, rsi - mov rax, rdi -.LBB5_4: + xor eax, eax + sub rdx, 4 + mov rcx, rdi + cmovb rcx, rax + shr rdx + test dil, 1 + cmove rax, rcx ret diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca index 3900a594613847..7ee51e0b8a406d 100644 --- a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1400 -Total Cycles: 405 -Total uOps: 1400 +Instructions: 900 +Total Cycles: 339 +Total uOps: 1100 Dispatch Width: 4 -uOps Per Cycle: 3.46 -IPC: 3.46 -Block RThroughput: 4.0 +uOps Per Cycle: 3.24 +IPC: 2.65 +Block RThroughput: 2.8 Instruction Info: @@ -18,19 +18,14 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 0 0.25 xor edx, edx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test dil, 1 - 1 1 1.00 jne .LBB5_4 - 1 1 0.33 cmp rsi, 4 - 1 1 1.00 jae .LBB5_3 - 1 1 0.33 mov edx, 1 - 1 0 0.25 xor eax, eax - 1 1 1.00 U ret - 1 1 0.33 add rsi, -4 - 1 1 0.50 shr rsi 1 1 0.33 mov rdx, rsi - 1 1 0.33 mov rax, rdi + 1 0 0.25 xor eax, eax + 1 1 0.33 sub rdx, 4 + 1 1 0.33 mov rcx, rdi + 2 2 0.67 cmovb rcx, rax + 1 1 0.50 shr rdx + 1 1 0.33 test dil, 1 + 2 2 0.67 cmove rax, rcx 1 1 1.00 U ret @@ -47,21 +42,16 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 3.99 3.99 - 4.02 - - + - - 3.33 3.33 - 3.34 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - - - - - - xor edx, edx - - - 0.01 0.98 - 0.01 - - mov eax, 0 - - - 0.98 0.02 - - - - test dil, 1 - - - - - - 1.00 - - jne .LBB5_4 - - - 0.02 0.98 - - - - cmp rsi, 4 - - - - - - 1.00 - - jae .LBB5_3 - - - 0.98 0.01 - 0.01 - - mov edx, 1 + - - 0.33 0.66 - 0.01 - - mov rdx, rsi - - - - - - - - xor eax, eax - - - - - - 1.00 - - ret - - - 0.01 0.99 - - - - add rsi, -4 - - - 1.00 - - - - - shr rsi - - - - 1.00 - - - - mov rdx, rsi - - - 0.99 0.01 - - - - mov rax, rdi + - - 0.01 - - 0.99 - - sub rdx, 4 + - - 0.66 0.34 - - - - mov rcx, rdi + - - 1.00 1.00 - - - - cmovb rcx, rax + - - 0.33 - - 0.67 - - shr rdx + - - - 0.33 - 0.67 - - test dil, 1 + - - 1.00 1.00 - - - - cmove rax, rcx - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 index 3e05f6023f38c2..68f21fe8018af0 100644 --- a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 @@ -1,23 +1,22 @@ bench_ref_from_suffix_dynamic_padding: lea eax, [rsi + rdi] test al, 3 - jne .LBB5_1 - movabs rax, 9223372036854775804 - and rax, rsi - cmp rax, 9 - jae .LBB5_3 -.LBB5_1: - xor eax, eax + setne al + cmp rsi, 12 + setb cl + or cl, al + je .LBB5_2 + xor edi, edi + mov rax, rdi ret -.LBB5_3: - add rax, -9 +.LBB5_2: + mov rax, rsi + or rax, 3 + add rax, -12 movabs rcx, -6148914691236517205 mul rcx shr rdx - lea rax, [rdx + 2*rdx] - sub rsi, rax - or rax, -4 - add rsi, rdi - add rax, rsi - add rax, -8 + and esi, 3 + add rdi, rsi + mov rax, rdi ret diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca index 73599d5b6aab84..c31e098aed4f35 100644 --- a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 Instructions: 2000 -Total Cycles: 682 +Total Cycles: 673 Total uOps: 2100 Dispatch Width: 4 -uOps Per Cycle: 3.08 -IPC: 2.93 +uOps Per Cycle: 3.12 +IPC: 2.97 Block RThroughput: 5.3 @@ -20,23 +20,23 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.50 lea eax, [rsi + rdi] 1 1 0.33 test al, 3 - 1 1 1.00 jne .LBB5_1 - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rax, rsi - 1 1 0.33 cmp rax, 9 - 1 1 1.00 jae .LBB5_3 - 1 0 0.25 xor eax, eax + 1 1 0.50 setne al + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setb cl + 1 1 0.33 or cl, al + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor edi, edi + 1 1 0.33 mov rax, rdi 1 1 1.00 U ret - 1 1 0.33 add rax, -9 + 1 1 0.33 mov rax, rsi + 1 1 0.33 or rax, 3 + 1 1 0.33 add rax, -12 1 1 0.33 movabs rcx, -6148914691236517205 2 4 1.00 mul rcx 1 1 0.50 shr rdx - 1 1 0.50 lea rax, [rdx + 2*rdx] - 1 1 0.33 sub rsi, rax - 1 1 0.33 or rax, -4 - 1 1 0.33 add rsi, rdi - 1 1 0.33 add rax, rsi - 1 1 0.33 add rax, -8 + 1 1 0.33 and esi, 3 + 1 1 0.33 add rdi, rsi + 1 1 0.33 mov rax, rdi 1 1 1.00 U ret @@ -53,27 +53,27 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.65 6.67 - 6.68 - - + - - 6.67 6.66 - 6.67 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.90 0.10 - - - - lea eax, [rsi + rdi] - - - 0.93 - - 0.07 - - test al, 3 - - - - - - 1.00 - - jne .LBB5_1 - - - 0.51 0.47 - 0.02 - - movabs rax, 9223372036854775804 - - - - - - 1.00 - - and rax, rsi - - - - 0.09 - 0.91 - - cmp rax, 9 - - - - - - 1.00 - - jae .LBB5_3 - - - - - - - - - xor eax, eax + - - 0.35 0.65 - - - - lea eax, [rsi + rdi] + - - 0.01 0.99 - - - - test al, 3 + - - 1.00 - - - - - setne al + - - 0.97 - - 0.03 - - cmp rsi, 12 + - - 0.66 - - 0.34 - - setb cl + - - - 1.00 - - - - or cl, al + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor edi, edi + - - 0.01 0.01 - 0.98 - - mov rax, rdi - - - - - 1.00 - - ret - - - 0.43 0.47 - 0.10 - - add rax, -9 - - - 0.42 0.39 - 0.19 - - movabs rcx, -6148914691236517205 + - - 0.01 0.99 - - - - mov rax, rsi + - - - 0.34 - 0.66 - - or rax, 3 + - - - 1.00 - - - - add rax, -12 + - - 0.66 0.31 - 0.03 - - movabs rcx, -6148914691236517205 - - 1.00 1.00 - - - - mul rcx - - - 0.69 - - 0.31 - - shr rdx - - - 0.54 0.46 - - - - lea rax, [rdx + 2*rdx] - - - 0.07 0.91 - 0.02 - - sub rsi, rax - - - 0.91 0.05 - 0.04 - - or rax, -4 - - - 0.08 0.90 - 0.02 - - add rsi, rdi - - - 0.09 0.91 - - - - add rax, rsi - - - 0.08 0.92 - - - - add rax, -8 + - - 1.00 - - - - - shr rdx + - - 0.67 0.01 - 0.32 - - and esi, 3 + - - 0.32 0.01 - 0.67 - - add rdi, rsi + - - 0.01 0.35 - 0.64 - - mov rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 index c3d10b5fc6853c..7d11849dd5c52e 100644 --- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 @@ -1,27 +1,21 @@ bench_ref_from_suffix_with_elems_dynamic_padding: - movabs rax, 3074457345618258598 - cmp rdx, rax - ja .LBB5_1 - lea r8d, [rsi + rdi] - xor ecx, ecx - mov eax, 0 - test r8b, 3 - je .LBB5_3 - mov rdx, rcx - ret + movabs rcx, 3074457345618258598 + cmp rdx, rcx + ja .LBB5_3 + mov rax, rdi + lea rcx, [rdx + 2*rdx] + or rcx, 3 + add rcx, 9 + add edi, esi + test dil, 3 + setne dil + sub rsi, rcx + setb cl + or cl, dil + je .LBB5_4 .LBB5_3: - lea rax, [rdx + 2*rdx] - or rax, 3 - add rax, 9 - sub rsi, rax - jae .LBB5_4 -.LBB5_1: xor eax, eax - mov edx, 1 ret .LBB5_4: - add rdi, rsi - mov rcx, rdx - mov rax, rdi - mov rdx, rcx + add rax, rsi ret diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca index 92e6280bb4cc8b..d7850d93bc1093 100644 --- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2300 -Total Cycles: 706 -Total uOps: 2300 +Instructions: 1800 +Total Cycles: 572 +Total uOps: 1800 Dispatch Width: 4 -uOps Per Cycle: 3.26 -IPC: 3.26 -Block RThroughput: 6.0 +uOps Per Cycle: 3.15 +IPC: 3.15 +Block RThroughput: 4.5 Instruction Info: @@ -18,28 +18,23 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 movabs rax, 3074457345618258598 - 1 1 0.33 cmp rdx, rax - 1 1 1.00 ja .LBB5_1 - 1 1 0.50 lea r8d, [rsi + rdi] - 1 0 0.25 xor ecx, ecx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test r8b, 3 - 1 1 1.00 je .LBB5_3 - 1 1 0.33 mov rdx, rcx - 1 1 1.00 U ret - 1 1 0.50 lea rax, [rdx + 2*rdx] - 1 1 0.33 or rax, 3 - 1 1 0.33 add rax, 9 - 1 1 0.33 sub rsi, rax - 1 1 1.00 jae .LBB5_4 + 1 1 0.33 movabs rcx, 3074457345618258598 + 1 1 0.33 cmp rdx, rcx + 1 1 1.00 ja .LBB5_3 + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea rcx, [rdx + 2*rdx] + 1 1 0.33 or rcx, 3 + 1 1 0.33 add rcx, 9 + 1 1 0.33 add edi, esi + 1 1 0.33 test dil, 3 + 1 1 0.50 setne dil + 1 1 0.33 sub rsi, rcx + 1 1 0.50 setb cl + 1 1 0.33 or cl, dil + 1 1 1.00 je .LBB5_4 1 0 0.25 xor eax, eax - 1 1 0.33 mov edx, 1 1 1 1.00 U ret - 1 1 0.33 add rdi, rsi - 1 1 0.33 mov rcx, rdx - 1 1 0.33 mov rax, rdi - 1 1 0.33 mov rdx, rcx + 1 1 0.33 add rax, rsi 1 1 1.00 U ret @@ -56,30 +51,25 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.99 7.00 - 7.01 - - + - - 5.66 5.66 - 5.68 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598 - - - 0.01 0.50 - 0.49 - - cmp rdx, rax - - - - - - 1.00 - - ja .LBB5_1 - - - - 1.00 - - - - lea r8d, [rsi + rdi] - - - - - - - - - xor ecx, ecx - - - 0.50 0.49 - 0.01 - - mov eax, 0 - - - 0.49 0.51 - - - - test r8b, 3 - - - - - - 1.00 - - je .LBB5_3 - - - 0.51 0.49 - - - - mov rdx, rcx - - - - - - 1.00 - - ret - - - 0.50 0.50 - - - - lea rax, [rdx + 2*rdx] - - - 1.00 - - - - - or rax, 3 - - - 1.00 - - - - - add rax, 9 - - - 0.99 0.01 - - - - sub rsi, rax - - - - - - 1.00 - - jae .LBB5_4 + - - 0.67 - - 0.33 - - movabs rcx, 3074457345618258598 + - - 0.34 - - 0.66 - - cmp rdx, rcx + - - - - - 1.00 - - ja .LBB5_3 + - - 0.33 0.01 - 0.66 - - mov rax, rdi + - - 0.33 0.67 - - - - lea rcx, [rdx + 2*rdx] + - - 0.01 0.99 - - - - or rcx, 3 + - - 0.01 0.99 - - - - add rcx, 9 + - - 0.99 - - 0.01 - - add edi, esi + - - 0.99 0.01 - - - - test dil, 3 + - - 0.99 - - 0.01 - - setne dil + - - - 1.00 - - - - sub rsi, rcx + - - 1.00 - - - - - setb cl + - - - 0.99 - 0.01 - - or cl, dil + - - - - - 1.00 - - je .LBB5_4 - - - - - - - - xor eax, eax - - - - 1.00 - - - - mov edx, 1 - - - - - 1.00 - - ret - - - 1.00 - - - - - add rdi, rsi - - - - 1.00 - - - - mov rcx, rdx - - - 0.99 0.01 - - - - mov rax, rdi - - - - 0.50 - 0.50 - - mov rdx, rcx + - - - 1.00 - - - - add rax, rsi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 index bdca57192455b7..7ee0278dbf684b 100644 --- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 @@ -1,23 +1,18 @@ bench_ref_from_suffix_with_elems_dynamic_size: - movabs rax, 4611686018427387901 - cmp rdx, rax - ja .LBB5_1 - lea r8d, [rsi + rdi] - xor ecx, ecx - mov eax, 0 - test r8b, 1 - jne .LBB5_5 - lea rax, [2*rdx + 4] - sub rsi, rax - jae .LBB5_4 -.LBB5_1: + movabs rcx, 4611686018427387901 + cmp rdx, rcx + ja .LBB5_3 + mov rax, rdi + lea rcx, [2*rdx + 4] + add edi, esi + sub rsi, rcx + setb cl + or cl, dil + test cl, 1 + je .LBB5_4 +.LBB5_3: xor eax, eax - mov edx, 1 ret .LBB5_4: - add rdi, rsi - mov rcx, rdx - mov rax, rdi -.LBB5_5: - mov rdx, rcx + add rax, rsi ret diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca index 6d9de0b3eb5cc1..882316de3d27de 100644 --- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1900 -Total Cycles: 571 -Total uOps: 1900 +Instructions: 1500 +Total Cycles: 472 +Total uOps: 1500 Dispatch Width: 4 -uOps Per Cycle: 3.33 -IPC: 3.33 -Block RThroughput: 5.0 +uOps Per Cycle: 3.18 +IPC: 3.18 +Block RThroughput: 4.0 Instruction Info: @@ -18,24 +18,20 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 movabs rax, 4611686018427387901 - 1 1 0.33 cmp rdx, rax - 1 1 1.00 ja .LBB5_1 - 1 1 0.50 lea r8d, [rsi + rdi] - 1 0 0.25 xor ecx, ecx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test r8b, 1 - 1 1 1.00 jne .LBB5_5 - 1 1 0.50 lea rax, [2*rdx + 4] - 1 1 0.33 sub rsi, rax - 1 1 1.00 jae .LBB5_4 + 1 1 0.33 movabs rcx, 4611686018427387901 + 1 1 0.33 cmp rdx, rcx + 1 1 1.00 ja .LBB5_3 + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea rcx, [2*rdx + 4] + 1 1 0.33 add edi, esi + 1 1 0.33 sub rsi, rcx + 1 1 0.50 setb cl + 1 1 0.33 or cl, dil + 1 1 0.33 test cl, 1 + 1 1 1.00 je .LBB5_4 1 0 0.25 xor eax, eax - 1 1 0.33 mov edx, 1 1 1 1.00 U ret - 1 1 0.33 add rdi, rsi - 1 1 0.33 mov rcx, rdx - 1 1 0.33 mov rax, rdi - 1 1 0.33 mov rdx, rcx + 1 1 0.33 add rax, rsi 1 1 1.00 U ret @@ -52,26 +48,22 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 5.66 5.66 - 5.68 - - + - - 4.66 4.66 - 4.68 - - Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901 - - - 0.01 0.99 - - - - cmp rdx, rax - - - - - - 1.00 - - ja .LBB5_1 - - - 0.99 0.01 - - - - lea r8d, [rsi + rdi] - - - - - - - - - xor ecx, ecx - - - 0.33 0.33 - 0.34 - - mov eax, 0 - - - 0.33 0.34 - 0.33 - - test r8b, 1 - - - - - - 1.00 - - jne .LBB5_5 - - - 0.34 0.66 - - - - lea rax, [2*rdx + 4] - - - - 1.00 - - - - sub rsi, rax - - - - - - 1.00 - - jae .LBB5_4 + - - 0.99 - - 0.01 - - movabs rcx, 4611686018427387901 + - - 1.00 - - - - - cmp rdx, rcx + - - - - - 1.00 - - ja .LBB5_3 + - - 0.63 0.02 - 0.35 - - mov rax, rdi + - - 0.35 0.65 - - - - lea rcx, [2*rdx + 4] + - - 0.65 0.03 - 0.32 - - add edi, esi + - - 0.03 0.97 - - - - sub rsi, rcx + - - 1.00 - - - - - setb cl + - - - 1.00 - - - - or cl, dil + - - 0.01 0.99 - - - - test cl, 1 + - - - - - 1.00 - - je .LBB5_4 - - - - - - - - xor eax, eax - - - 1.00 - - - - - mov edx, 1 - - - - - 1.00 - - ret - - - - 1.00 - - - - add rdi, rsi - - - 1.00 - - - - - mov rcx, rdx - - - 0.32 0.68 - - - - mov rax, rdi - - - 0.68 0.32 - - - - mov rdx, rcx + - - - 1.00 - - - - add rax, rsi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs new file mode 100644 index 00000000000000..8d4e2639459693 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_dynamic_padding<'a>( + source: &'a Ref<&[u8], format::LocoPacket>, +) -> &'a [u8] { + Ref::bytes(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..a44c1510fe0201 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_dynamic_padding: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs new file mode 100644 index 00000000000000..ef7dcf83989db3 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs @@ -0,0 +1,17 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_dynamic_size<'a>(source: &'a Ref<&[u8], format::LocoPacket>) -> &'a [u8] { + Ref::bytes(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64 new file mode 100644 index 00000000000000..6ac3ad2f1fefdc --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_dynamic_size: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs new file mode 100644 index 00000000000000..e99b6704bd9b28 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_mut_dynamic_padding<'a>( + source: &'a mut Ref<&mut [u8], format::LocoPacket>, +) -> &'a mut [u8] { + Ref::bytes_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..ebc6ee04941e44 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_mut_dynamic_padding: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs new file mode 100644 index 00000000000000..badf9de19f0617 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_mut_dynamic_size<'a>( + source: &'a mut Ref<&mut [u8], format::LocoPacket>, +) -> &'a mut [u8] { + Ref::bytes_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64 new file mode 100644 index 00000000000000..5d97d01e51e28d --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_mut_dynamic_size: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs new file mode 100644 index 00000000000000..1a49c2492a41ea --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_mut_static_size<'a>( + source: &'a mut Ref<&mut [u8], format::LocoPacket>, +) -> &'a mut [u8] { + Ref::bytes_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64 new file mode 100644 index 00000000000000..3411bd45e96aa9 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_mut_static_size: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.rs b/rust/zerocopy/benches/ref_type_bytes_static_size.rs new file mode 100644 index 00000000000000..b9c3fe5b7bb53e --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_static_size.rs @@ -0,0 +1,17 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_bytes_static_size<'a>(source: &'a Ref<&[u8], format::LocoPacket>) -> &'a [u8] { + Ref::bytes(source) +} diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64 new file mode 100644 index 00000000000000..10a024acaafd50 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64 @@ -0,0 +1,4 @@ +bench_ref_type_bytes_static_size: + mov rax, qword ptr [rdi] + mov rdx, qword ptr [rdi + 8] + ret diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca new file mode 100644 index 00000000000000..a7f8947c04133b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca @@ -0,0 +1,45 @@ +Iterations: 100 +Instructions: 300 +Total Cycles: 107 +Total uOps: 300 + +Dispatch Width: 4 +uOps Per Cycle: 2.80 +IPC: 2.80 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rdx, qword ptr [rdi + 8] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 1.00 1.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - - 1.00 mov rax, qword ptr [rdi] + - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs new file mode 100644 index 00000000000000..c0ebe56ccc7584 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_deref_dynamic_padding<'a>( + source: &'a Ref<&[u8], format::LocoPacket>, +) -> &'a format::LocoPacket { + core::ops::Deref::deref(source) +} diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..765c5305d86d69 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64 @@ -0,0 +1,29 @@ +bench_ref_type_deref_dynamic_padding: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 3074457345618258599 + cmp rdx, rdi + jae .LBB8_3 + test al, 3 + jne .LBB8_2 + lea rdi, [rdx + 2*rdx] + or rdi, 3 + add rdi, 9 + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.7] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.9] + mov rdx, rsp + mov esi, 58 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..bea2c8a20ea782 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca @@ -0,0 +1,91 @@ +Iterations: 100 +Instructions: 2600 +Total Cycles: 1812 +Total uOps: 2900 + +Dispatch Width: 4 +uOps Per Cycle: 1.60 +IPC: 1.43 +Block RThroughput: 7.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 3074457345618258599 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 3 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [rdx + 2*rdx] + 1 1 0.33 or rdi, 3 + 1 1 0.33 add rdi, 9 + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.9] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 58 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.41 6.42 4.00 6.17 3.99 4.01 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.54 0.46 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - - 0.49 0.51 mov rcx, qword ptr [rdi + 8] + - - 0.52 0.47 - 0.01 - - mov esi, 1 + - - 0.47 0.53 - - - - movabs rdi, 3074457345618258599 + - - 0.47 0.53 - - - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.54 0.44 - 0.02 - - test al, 3 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.44 0.56 - - - - lea rdi, [rdx + 2*rdx] + - - 0.48 0.52 - - - - or rdi, 3 + - - 0.48 0.48 - 0.04 - - add rdi, 9 + - - 0.51 0.39 - 0.10 - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.53 0.47 - - - - sub rsp, 24 + - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.48 0.52 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp + 16], rcx + - - 0.49 0.51 - - - - lea rdi, [rip + .Lanon.HASH.7] + - - 0.46 0.54 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.54 0.46 - - - - lea r8, [rip + .Lanon.HASH.9] + - - 0.04 0.06 - 0.90 - - mov rdx, rsp + - - 0.44 0.46 - 0.10 - - mov esi, 58 + - - - - 1.00 1.00 0.94 1.06 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs b/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs new file mode 100644 index 00000000000000..adcf89c2250840 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_deref_dynamic_size<'a>( + source: &'a Ref<&[u8], format::LocoPacket>, +) -> &'a format::LocoPacket { + core::ops::Deref::deref(source) +} diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64 new file mode 100644 index 00000000000000..60ca3b508a7e15 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_deref_dynamic_size: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 4611686018427387902 + cmp rdx, rdi + jae .LBB8_3 + test al, 1 + jne .LBB8_2 + lea rdi, [2*rdx + 4] + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.7] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.9] + mov rdx, rsp + mov esi, 58 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..66a8f8575acd36 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 1647 +Total uOps: 2700 + +Dispatch Width: 4 +uOps Per Cycle: 1.64 +IPC: 1.46 +Block RThroughput: 6.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 4611686018427387902 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 1 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [2*rdx + 4] + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.9] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 58 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 5.15 5.74 4.00 6.11 3.98 4.02 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.72 0.28 mov rax, qword ptr [rdi] + - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8] + - - 0.83 0.16 - 0.01 - - mov esi, 1 + - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902 + - - 0.80 0.04 - 0.16 - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.13 0.74 - 0.13 - - test al, 1 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4] + - - 0.09 0.91 - - - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.29 0.13 - 0.58 - - sub rsp, 24 + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx + - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.7] + - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.9] + - - 0.09 0.68 - 0.23 - - mov rdx, rsp + - - 0.16 0.84 - - - - mov esi, 58 + - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs new file mode 100644 index 00000000000000..263f2c575379cd --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_deref_mut_dynamic_size<'a>( + source: &'a mut Ref<&mut [u8], format::LocoPacket>, +) -> &'a mut format::LocoPacket { + core::ops::DerefMut::deref_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64 new file mode 100644 index 00000000000000..5ae0ae69efad47 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_deref_mut_dynamic_size: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 4611686018427387902 + cmp rdx, rdi + jae .LBB8_3 + test al, 1 + jne .LBB8_2 + lea rdi, [2*rdx + 4] + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.7] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.9] + mov rdx, rsp + mov esi, 65 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..ed3a2bc247013e --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 1647 +Total uOps: 2700 + +Dispatch Width: 4 +uOps Per Cycle: 1.64 +IPC: 1.46 +Block RThroughput: 6.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 4611686018427387902 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 1 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [2*rdx + 4] + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.9] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 65 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 5.15 5.74 4.00 6.11 3.98 4.02 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.72 0.28 mov rax, qword ptr [rdi] + - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8] + - - 0.83 0.16 - 0.01 - - mov esi, 1 + - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902 + - - 0.80 0.04 - 0.16 - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.13 0.74 - 0.13 - - test al, 1 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4] + - - 0.09 0.91 - - - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.29 0.13 - 0.58 - - sub rsp, 24 + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx + - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.7] + - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.9] + - - 0.09 0.68 - 0.23 - - mov rdx, rsp + - - 0.16 0.84 - - - - mov esi, 65 + - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs b/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs new file mode 100644 index 00000000000000..4f85a031c845f0 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_deref_mut_static_size<'a>( + source: &'a mut Ref<&mut [u8], format::LocoPacket>, +) -> &'a mut format::LocoPacket { + core::ops::DerefMut::deref_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64 new file mode 100644 index 00000000000000..259c39646bb383 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64 @@ -0,0 +1,3 @@ +bench_ref_type_deref_mut_static_size: + mov rax, qword ptr [rdi] + ret diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca new file mode 100644 index 00000000000000..2de0e6f9791c9f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca @@ -0,0 +1,43 @@ +Iterations: 100 +Instructions: 200 +Total Cycles: 103 +Total uOps: 200 + +Dispatch Width: 4 +uOps Per Cycle: 1.94 +IPC: 1.94 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 0.50 0.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.rs b/rust/zerocopy/benches/ref_type_deref_static_size.rs new file mode 100644 index 00000000000000..60397dc28580f9 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_static_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_deref_static_size<'a>( + source: &'a Ref<&[u8], format::LocoPacket>, +) -> &'a format::LocoPacket { + core::ops::Deref::deref(source) +} diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64 new file mode 100644 index 00000000000000..2d4368dd61dfc3 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64 @@ -0,0 +1,3 @@ +bench_ref_type_deref_static_size: + mov rax, qword ptr [rdi] + ret diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca new file mode 100644 index 00000000000000..2de0e6f9791c9f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca @@ -0,0 +1,43 @@ +Iterations: 100 +Instructions: 200 +Total Cycles: 103 +Total uOps: 200 + +Dispatch Width: 4 +uOps Per Cycle: 1.94 +IPC: 1.94 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 0.50 0.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs new file mode 100644 index 00000000000000..d11cae87a1df38 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_bytes_dynamic_padding( + source: &[u8], +) -> Option> { + Ref::from_bytes(source).ok() +} diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..9f2beef51011e0 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64 @@ -0,0 +1,25 @@ +bench_ref_type_from_bytes_dynamic_padding: + cmp rdx, 12 + setae al + mov ecx, edx + or ecx, esi + test cl, 3 + sete cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rdi], rsi + mov rax, rdi + ret +.LBB5_2: + lea rax, [rdx - 9] + movabs rcx, -6148914691236517205 + mov r8, rdx + mul rcx + shr rdx + mov qword ptr [rdi + 8], r8 + mov qword ptr [rdi + 16], rdx + mov qword ptr [rdi], rsi + mov rax, rdi + ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..e239f2fbe9e768 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca @@ -0,0 +1,85 @@ +Iterations: 100 +Instructions: 2300 +Total Cycles: 909 +Total uOps: 2400 + +Dispatch Width: 4 +uOps Per Cycle: 2.64 +IPC: 2.53 +Block RThroughput: 6.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 cmp rdx, 12 + 1 1 0.50 setae al + 1 1 0.33 mov ecx, edx + 1 1 0.33 or ecx, esi + 1 1 0.33 test cl, 3 + 1 1 0.50 sete cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + 1 1 0.50 lea rax, [rdx - 9] + 1 1 0.33 movabs rcx, -6148914691236517205 + 1 1 0.33 mov r8, rdx + 2 4 1.00 mul rcx + 1 1 0.50 shr rdx + 1 1 1.00 * mov qword ptr [rdi + 8], r8 + 1 1 1.00 * mov qword ptr [rdi + 16], rdx + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.49 5.99 4.00 6.52 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - 0.99 - 0.01 - - cmp rdx, 12 + - - 0.53 - - 0.47 - - setae al + - - 0.52 0.01 - 0.47 - - mov ecx, edx + - - - 0.99 - 0.01 - - or ecx, esi + - - - 0.51 - 0.49 - - test cl, 3 + - - 0.52 - - 0.48 - - sete cl + - - 0.47 0.48 - 0.05 - - and cl, al + - - 0.48 0.48 - 0.04 - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi + - - 0.03 0.48 - 0.49 - - mov rax, rdi + - - - - - 1.00 - - ret + - - 0.97 0.03 - - - - lea rax, [rdx - 9] + - - 0.49 0.51 - - - - movabs rcx, -6148914691236517205 + - - 0.47 0.01 - 0.52 - - mov r8, rdx + - - 1.00 1.00 - - - - mul rcx + - - 1.00 - - - - - shr rdx + - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], r8 + - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi + - - 0.01 0.50 - 0.49 - - mov rax, rdi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs new file mode 100644 index 00000000000000..cf0b278eaf0964 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs @@ -0,0 +1,17 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_bytes_dynamic_size(source: &[u8]) -> Option> { + Ref::from_bytes(source).ok() +} diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64 new file mode 100644 index 00000000000000..c50dfc333bd6ae --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64 @@ -0,0 +1,21 @@ +bench_ref_type_from_bytes_dynamic_size: + mov rax, rdi + cmp rdx, 4 + setae cl + mov edi, edx + or edi, esi + test dil, 1 + sete dil + and dil, cl + cmp dil, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rax], rsi + ret +.LBB5_2: + lea rcx, [rdx - 4] + shr rcx + mov qword ptr [rax + 8], rdx + mov qword ptr [rax + 16], rcx + mov qword ptr [rax], rsi + ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..92d98788cc09a3 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca @@ -0,0 +1,77 @@ +Iterations: 100 +Instructions: 1900 +Total Cycles: 482 +Total uOps: 1900 + +Dispatch Width: 4 +uOps Per Cycle: 3.94 +IPC: 3.94 +Block RThroughput: 4.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 cmp rdx, 4 + 1 1 0.50 setae cl + 1 1 0.33 mov edi, edx + 1 1 0.33 or edi, esi + 1 1 0.33 test dil, 1 + 1 1 0.50 sete dil + 1 1 0.33 and dil, cl + 1 1 0.33 cmp dil, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + 1 1 0.50 lea rcx, [rdx - 4] + 1 1 0.50 shr rcx + 1 1 1.00 * mov qword ptr [rax + 8], rdx + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 4.66 4.66 4.00 4.68 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.40 0.43 - 0.17 - - mov rax, rdi + - - 0.13 0.83 - 0.04 - - cmp rdx, 4 + - - 0.67 - - 0.33 - - setae cl + - - 0.05 0.79 - 0.16 - - mov edi, edx + - - 0.36 0.52 - 0.12 - - or edi, esi + - - 0.39 0.52 - 0.09 - - test dil, 1 + - - 0.98 - - 0.02 - - sete dil + - - 0.42 0.49 - 0.09 - - and dil, cl + - - 0.43 0.41 - 0.16 - - cmp dil, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi + - - - - - 1.00 - - ret + - - 0.33 0.67 - - - - lea rcx, [rdx - 4] + - - 0.50 - - 0.50 - - shr rcx + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs new file mode 100644 index 00000000000000..6a2ced8aad89a7 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs @@ -0,0 +1,17 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_bytes_static_size(source: &[u8]) -> Option> { + Ref::from_bytes(source).ok() +} diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64 new file mode 100644 index 00000000000000..59c3862eb5693c --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64 @@ -0,0 +1,15 @@ +bench_ref_type_from_bytes_static_size: + mov rax, rdi + cmp rdx, 6 + setne cl + or cl, sil + test cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rax], rsi + ret +.LBB5_2: + mov qword ptr [rax + 8], 6 + mov qword ptr [rax + 16], 0 + mov qword ptr [rax], rsi + ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca new file mode 100644 index 00000000000000..286354a1a014f9 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca @@ -0,0 +1,65 @@ +Iterations: 100 +Instructions: 1300 +Total Cycles: 404 +Total uOps: 1300 + +Dispatch Width: 4 +uOps Per Cycle: 3.22 +IPC: 3.22 +Block RThroughput: 4.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 cmp rdx, 6 + 1 1 0.50 setne cl + 1 1 0.33 or cl, sil + 1 1 0.33 test cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax + 8], 6 + 1 1 1.00 * mov qword ptr [rax + 16], 0 + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 2.55 2.34 4.00 3.11 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.81 0.18 - 0.01 - - mov rax, rdi + - - 0.56 0.43 - 0.01 - - cmp rdx, 6 + - - 1.00 - - - - - setne cl + - - - 0.99 - 0.01 - - or cl, sil + - - 0.18 0.74 - 0.08 - - test cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6 + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0 + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs new file mode 100644 index 00000000000000..5d3ee937c2b37a --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs @@ -0,0 +1,20 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_bytes_with_elems_dynamic_padding( + source: &[u8], + count: usize, +) -> Option> { + Ref::from_bytes_with_elems(source, count).ok() +} diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..0eb8287a145eeb --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64 @@ -0,0 +1,26 @@ +bench_ref_type_from_bytes_with_elems_dynamic_padding: + mov rax, rdi + movabs rdi, 6148914691236517201 + cmp rcx, rdi + ja .LBB5_3 + lea rdi, [rcx + 2*rcx] + or rdi, 3 + add rdi, 9 + cmp rdx, rdi + sete dil + movabs r8, 3074457345618258599 + cmp rcx, r8 + setb r8b + and r8b, dil + cmp r8b, 1 + jne .LBB5_3 + test sil, 3 + je .LBB5_4 +.LBB5_3: + mov qword ptr [rax], 0 + ret +.LBB5_4: + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdx + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..19928fdca6b8b1 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca @@ -0,0 +1,85 @@ +Iterations: 100 +Instructions: 2300 +Total Cycles: 640 +Total uOps: 2300 + +Dispatch Width: 4 +uOps Per Cycle: 3.59 +IPC: 3.59 +Block RThroughput: 5.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 movabs rdi, 6148914691236517201 + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 ja .LBB5_3 + 1 1 0.50 lea rdi, [rcx + 2*rcx] + 1 1 0.33 or rdi, 3 + 1 1 0.33 add rdi, 9 + 1 1 0.33 cmp rdx, rdi + 1 1 0.50 sete dil + 1 1 0.33 movabs r8, 3074457345618258599 + 1 1 0.33 cmp rcx, r8 + 1 1 0.50 setb r8b + 1 1 0.33 and r8b, dil + 1 1 0.33 cmp r8b, 1 + 1 1 1.00 jne .LBB5_3 + 1 1 0.33 test sil, 3 + 1 1 1.00 je .LBB5_4 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdx + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.32 6.33 4.00 6.35 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.99 - - 0.01 - - mov rax, rdi + - - 0.66 0.34 - - - - movabs rdi, 6148914691236517201 + - - 0.34 0.66 - - - - cmp rcx, rdi + - - - - - 1.00 - - ja .LBB5_3 + - - 0.66 0.34 - - - - lea rdi, [rcx + 2*rcx] + - - 0.01 0.99 - - - - or rdi, 3 + - - 0.01 0.99 - - - - add rdi, 9 + - - 0.01 0.99 - - - - cmp rdx, rdi + - - 0.99 - - 0.01 - - sete dil + - - 0.35 0.01 - 0.64 - - movabs r8, 3074457345618258599 + - - 0.99 - - 0.01 - - cmp rcx, r8 + - - 0.99 - - 0.01 - - setb r8b + - - - 1.00 - - - - and r8b, dil + - - - 1.00 - - - - cmp r8b, 1 + - - - - - 1.00 - - jne .LBB5_3 + - - 0.32 0.01 - 0.67 - - test sil, 3 + - - - - - 1.00 - - je .LBB5_4 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs new file mode 100644 index 00000000000000..0ea631ab0c48b6 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs @@ -0,0 +1,20 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_bytes_with_elems_dynamic_size( + source: &[u8], + count: usize, +) -> Option> { + Ref::from_bytes_with_elems(source, count).ok() +} diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64 new file mode 100644 index 00000000000000..67cc5b58dd14d0 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64 @@ -0,0 +1,28 @@ +bench_ref_type_from_bytes_with_elems_dynamic_size: + mov rax, rdi + lea rdi, [rcx + rcx] + test rcx, rcx + setns r8b + cmp rdi, -5 + setb dil + and dil, r8b + lea r8, [2*rcx + 4] + cmp rdx, r8 + sete r8b + movabs r9, 4611686018427387902 + cmp rcx, r9 + setb r9b + and r9b, r8b + and r9b, dil + cmp r9b, 1 + je .LBB5_1 +.LBB5_2: + mov qword ptr [rax], 0 + ret +.LBB5_1: + test sil, 1 + jne .LBB5_2 + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdx + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..cc99f6a4c6f32c --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca @@ -0,0 +1,89 @@ +Iterations: 100 +Instructions: 2500 +Total Cycles: 706 +Total uOps: 2500 + +Dispatch Width: 4 +uOps Per Cycle: 3.54 +IPC: 3.54 +Block RThroughput: 6.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea rdi, [rcx + rcx] + 1 1 0.33 test rcx, rcx + 1 1 0.50 setns r8b + 1 1 0.33 cmp rdi, -5 + 1 1 0.50 setb dil + 1 1 0.33 and dil, r8b + 1 1 0.50 lea r8, [2*rcx + 4] + 1 1 0.33 cmp rdx, r8 + 1 1 0.50 sete r8b + 1 1 0.33 movabs r9, 4611686018427387902 + 1 1 0.33 cmp rcx, r9 + 1 1 0.50 setb r9b + 1 1 0.33 and r9b, r8b + 1 1 0.33 and r9b, dil + 1 1 0.33 cmp r9b, 1 + 1 1 1.00 je .LBB5_1 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 0.33 test sil, 1 + 1 1 1.00 jne .LBB5_2 + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdx + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.99 7.00 4.00 7.01 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.08 - - 0.92 - - mov rax, rdi + - - 0.99 0.01 - - - - lea rdi, [rcx + rcx] + - - 0.92 0.01 - 0.07 - - test rcx, rcx + - - 0.99 - - 0.01 - - setns r8b + - - 0.01 0.01 - 0.98 - - cmp rdi, -5 + - - 0.98 - - 0.02 - - setb dil + - - - 0.99 - 0.01 - - and dil, r8b + - - 0.01 0.99 - - - - lea r8, [2*rcx + 4] + - - 0.99 0.01 - - - - cmp rdx, r8 + - - 1.00 - - - - - sete r8b + - - 0.01 - - 0.99 - - movabs r9, 4611686018427387902 + - - - 1.00 - - - - cmp rcx, r9 + - - 0.99 - - 0.01 - - setb r9b + - - 0.01 0.99 - - - - and r9b, r8b + - - 0.01 0.99 - - - - and r9b, dil + - - - 1.00 - - - - cmp r9b, 1 + - - - - - 1.00 - - je .LBB5_1 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - 1.00 - - - - test sil, 1 + - - - - - 1.00 - - jne .LBB5_2 + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs new file mode 100644 index 00000000000000..7826991fc546e2 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs @@ -0,0 +1,22 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_prefix_dynamic_padding( + source: &[u8], +) -> Option> { + match Ref::from_prefix(source) { + Ok((packet, _rest)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..a549e33379850a --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64 @@ -0,0 +1,26 @@ +bench_ref_type_from_prefix_dynamic_padding: + test sil, 3 + sete al + cmp rdx, 12 + setae cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rdi], rsi + mov rax, rdi + ret +.LBB5_2: + movabs rcx, 9223372036854775804 + and rcx, rdx + or rdx, 3 + add rdx, -12 + movabs r8, -6148914691236517205 + mov rax, rdx + mul r8 + shr rdx + mov qword ptr [rdi + 8], rcx + mov qword ptr [rdi + 16], rdx + mov qword ptr [rdi], rsi + mov rax, rdi + ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..d85fe71ca66f30 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 907 +Total uOps: 2500 + +Dispatch Width: 4 +uOps Per Cycle: 2.76 +IPC: 2.65 +Block RThroughput: 6.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 test sil, 3 + 1 1 0.50 sete al + 1 1 0.33 cmp rdx, 12 + 1 1 0.50 setae cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + 1 1 0.33 movabs rcx, 9223372036854775804 + 1 1 0.33 and rcx, rdx + 1 1 0.33 or rdx, 3 + 1 1 0.33 add rdx, -12 + 1 1 0.33 movabs r8, -6148914691236517205 + 1 1 0.33 mov rax, rdx + 2 4 1.00 mul r8 + 1 1 0.50 shr rdx + 1 1 1.00 * mov qword ptr [rdi + 8], rcx + 1 1 1.00 * mov qword ptr [rdi + 16], rdx + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.96 6.07 4.00 6.97 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.92 0.07 - 0.01 - - test sil, 3 + - - 0.98 - - 0.02 - - sete al + - - 0.03 0.97 - - - - cmp rdx, 12 + - - 0.99 - - 0.01 - - setae cl + - - 0.03 0.94 - 0.03 - - and cl, al + - - 0.06 0.94 - - - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi + - - 0.01 0.05 - 0.94 - - mov rax, rdi + - - - - - 1.00 - - ret + - - 0.92 0.08 - - - - movabs rcx, 9223372036854775804 + - - 0.96 0.01 - 0.03 - - and rcx, rdx + - - - 0.03 - 0.97 - - or rdx, 3 + - - 0.01 0.96 - 0.03 - - add rdx, -12 + - - 0.03 0.06 - 0.91 - - movabs r8, -6148914691236517205 + - - 0.94 0.03 - 0.03 - - mov rax, rdx + - - 1.00 1.00 - - - - mul r8 + - - 0.05 - - 0.95 - - shr rdx + - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], rcx + - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi + - - 0.03 0.93 - 0.04 - - mov rax, rdi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs new file mode 100644 index 00000000000000..131a1210365628 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs @@ -0,0 +1,22 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_prefix_dynamic_size( + source: &[u8], +) -> Option> { + match Ref::from_prefix(source) { + Ok((packet, _rest)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64 new file mode 100644 index 00000000000000..fc050aac263710 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64 @@ -0,0 +1,19 @@ +bench_ref_type_from_prefix_dynamic_size: + mov rax, rdi + cmp rdx, 4 + setb cl + or cl, sil + test cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rax], rsi + ret +.LBB5_2: + movabs rcx, 9223372036854775806 + and rcx, rdx + add rdx, -4 + shr rdx + mov qword ptr [rax + 8], rcx + mov qword ptr [rax + 16], rdx + mov qword ptr [rax], rsi + ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..a249f0305291ce --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca @@ -0,0 +1,73 @@ +Iterations: 100 +Instructions: 1700 +Total Cycles: 431 +Total uOps: 1700 + +Dispatch Width: 4 +uOps Per Cycle: 3.94 +IPC: 3.94 +Block RThroughput: 4.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 cmp rdx, 4 + 1 1 0.50 setb cl + 1 1 0.33 or cl, sil + 1 1 0.33 test cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + 1 1 0.33 movabs rcx, 9223372036854775806 + 1 1 0.33 and rcx, rdx + 1 1 0.33 add rdx, -4 + 1 1 0.50 shr rdx + 1 1 1.00 * mov qword ptr [rax + 8], rcx + 1 1 1.00 * mov qword ptr [rax + 16], rdx + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 3.99 3.75 4.00 4.26 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - 0.99 - 0.01 - - mov rax, rdi + - - 0.74 0.26 - - - - cmp rdx, 4 + - - 1.00 - - - - - setb cl + - - 0.25 0.74 - 0.01 - - or cl, sil + - - 0.01 0.97 - 0.02 - - test cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi + - - - - - 1.00 - - ret + - - 0.48 0.52 - - - - movabs rcx, 9223372036854775806 + - - 0.73 0.27 - - - - and rcx, rdx + - - 0.02 - - 0.98 - - add rdx, -4 + - - 0.76 - - 0.24 - - shr rdx + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rcx + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs new file mode 100644 index 00000000000000..c149fd5088ee2a --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs @@ -0,0 +1,20 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_prefix_static_size(source: &[u8]) -> Option> { + match Ref::from_prefix(source) { + Ok((packet, _rest)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64 new file mode 100644 index 00000000000000..420fd6f3b4bd2b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64 @@ -0,0 +1,15 @@ +bench_ref_type_from_prefix_static_size: + mov rax, rdi + cmp rdx, 6 + setb cl + or cl, sil + test cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rax], rsi + ret +.LBB5_2: + mov qword ptr [rax + 8], 6 + mov qword ptr [rax + 16], 0 + mov qword ptr [rax], rsi + ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca new file mode 100644 index 00000000000000..6b65f303a78698 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca @@ -0,0 +1,65 @@ +Iterations: 100 +Instructions: 1300 +Total Cycles: 404 +Total uOps: 1300 + +Dispatch Width: 4 +uOps Per Cycle: 3.22 +IPC: 3.22 +Block RThroughput: 4.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 cmp rdx, 6 + 1 1 0.50 setb cl + 1 1 0.33 or cl, sil + 1 1 0.33 test cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax + 8], 6 + 1 1 1.00 * mov qword ptr [rax + 16], 0 + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 2.55 2.34 4.00 3.11 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.81 0.18 - 0.01 - - mov rax, rdi + - - 0.56 0.43 - 0.01 - - cmp rdx, 6 + - - 1.00 - - - - - setb cl + - - - 0.99 - 0.01 - - or cl, sil + - - 0.18 0.74 - 0.08 - - test cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6 + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0 + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs new file mode 100644 index 00000000000000..a4fe5a6c86ca39 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs @@ -0,0 +1,23 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_prefix_with_elems_dynamic_padding( + source: &[u8], + count: usize, +) -> Option> { + match Ref::from_prefix_with_elems(source, count) { + Ok((packet, _rest)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..fb9d6bfaac270f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64 @@ -0,0 +1,26 @@ +bench_ref_type_from_prefix_with_elems_dynamic_padding: + mov rax, rdi + movabs rdi, 6148914691236517201 + cmp rcx, rdi + ja .LBB5_4 + lea rdi, [rcx + 2*rcx] + or rdi, 3 + add rdi, 9 + cmp rdi, rdx + setbe dl + movabs r8, 3074457345618258599 + cmp rcx, r8 + setb r8b + and r8b, dl + cmp r8b, 1 + jne .LBB5_4 + test sil, 3 + je .LBB5_3 +.LBB5_4: + mov qword ptr [rax], 0 + ret +.LBB5_3: + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdi + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..0ef4c84142b017 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca @@ -0,0 +1,85 @@ +Iterations: 100 +Instructions: 2300 +Total Cycles: 674 +Total uOps: 2400 + +Dispatch Width: 4 +uOps Per Cycle: 3.56 +IPC: 3.41 +Block RThroughput: 6.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 movabs rdi, 6148914691236517201 + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 ja .LBB5_4 + 1 1 0.50 lea rdi, [rcx + 2*rcx] + 1 1 0.33 or rdi, 3 + 1 1 0.33 add rdi, 9 + 1 1 0.33 cmp rdi, rdx + 2 2 1.00 setbe dl + 1 1 0.33 movabs r8, 3074457345618258599 + 1 1 0.33 cmp rcx, r8 + 1 1 0.50 setb r8b + 1 1 0.33 and r8b, dl + 1 1 0.33 cmp r8b, 1 + 1 1 1.00 jne .LBB5_4 + 1 1 0.33 test sil, 3 + 1 1 1.00 je .LBB5_3 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdi + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.66 6.65 4.00 6.69 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.01 0.98 - 0.01 - - mov rax, rdi + - - 0.01 0.99 - - - - movabs rdi, 6148914691236517201 + - - 0.02 0.98 - - - - cmp rcx, rdi + - - - - - 1.00 - - ja .LBB5_4 + - - 0.02 0.98 - - - - lea rdi, [rcx + 2*rcx] + - - 0.68 0.32 - - - - or rdi, 3 + - - 0.98 0.02 - - - - add rdi, 9 + - - 0.97 0.03 - - - - cmp rdi, rdx + - - 1.96 - - 0.04 - - setbe dl + - - 0.02 0.98 - - - - movabs r8, 3074457345618258599 + - - 0.02 0.34 - 0.64 - - cmp rcx, r8 + - - 0.02 - - 0.98 - - setb r8b + - - 0.97 0.02 - 0.01 - - and r8b, dl + - - 0.97 0.02 - 0.01 - - cmp r8b, 1 + - - - - - 1.00 - - jne .LBB5_4 + - - 0.01 0.99 - - - - test sil, 3 + - - - - - 1.00 - - je .LBB5_3 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs new file mode 100644 index 00000000000000..b0911e8fd8e73d --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs @@ -0,0 +1,23 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_prefix_with_elems_dynamic_size( + source: &[u8], + count: usize, +) -> Option> { + match Ref::from_prefix_with_elems(source, count) { + Ok((packet, _rest)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64 new file mode 100644 index 00000000000000..20ba73bd552eb1 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_from_prefix_with_elems_dynamic_size: + mov rax, rdi + lea rdi, [rcx + rcx] + test rcx, rcx + sets r8b + cmp rdi, -5 + setae r9b + or r9b, r8b + jne .LBB5_3 + add rdi, 4 + cmp rdi, rdx + ja .LBB5_3 + movabs rdx, 4611686018427387902 + cmp rcx, rdx + setb dl + test sil, 1 + sete r8b + test dl, r8b + jne .LBB5_4 +.LBB5_3: + mov qword ptr [rax], 0 + ret +.LBB5_4: + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdi + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..5600d4ea6f6981 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 672 +Total uOps: 2400 + +Dispatch Width: 4 +uOps Per Cycle: 3.57 +IPC: 3.57 +Block RThroughput: 6.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea rdi, [rcx + rcx] + 1 1 0.33 test rcx, rcx + 1 1 0.50 sets r8b + 1 1 0.33 cmp rdi, -5 + 1 1 0.50 setae r9b + 1 1 0.33 or r9b, r8b + 1 1 1.00 jne .LBB5_3 + 1 1 0.33 add rdi, 4 + 1 1 0.33 cmp rdi, rdx + 1 1 1.00 ja .LBB5_3 + 1 1 0.33 movabs rdx, 4611686018427387902 + 1 1 0.33 cmp rcx, rdx + 1 1 0.50 setb dl + 1 1 0.33 test sil, 1 + 1 1 0.50 sete r8b + 1 1 0.33 test dl, r8b + 1 1 1.00 jne .LBB5_4 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdi + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.66 6.65 4.00 6.69 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.03 0.96 - 0.01 - - mov rax, rdi + - - 0.64 0.36 - - - - lea rdi, [rcx + rcx] + - - 0.34 0.66 - - - - test rcx, rcx + - - 0.99 - - 0.01 - - sets r8b + - - 0.66 0.34 - - - - cmp rdi, -5 + - - 0.67 - - 0.33 - - setae r9b + - - 0.01 0.99 - - - - or r9b, r8b + - - - - - 1.00 - - jne .LBB5_3 + - - - 0.99 - 0.01 - - add rdi, 4 + - - 0.66 0.33 - 0.01 - - cmp rdi, rdx + - - - - - 1.00 - - ja .LBB5_3 + - - - 0.01 - 0.99 - - movabs rdx, 4611686018427387902 + - - 0.33 0.34 - 0.33 - - cmp rcx, rdx + - - 1.00 - - - - - setb dl + - - 0.33 0.67 - - - - test sil, 1 + - - 1.00 - - - - - sete r8b + - - - 1.00 - - - - test dl, r8b + - - - - - 1.00 - - jne .LBB5_4 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs new file mode 100644 index 00000000000000..894b7d875b42b1 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs @@ -0,0 +1,22 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_suffix_dynamic_padding( + source: &[u8], +) -> Option> { + match Ref::from_suffix(source) { + Ok((_rest, packet)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..4a0f5e88612e58 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64 @@ -0,0 +1,31 @@ +bench_ref_type_from_suffix_dynamic_padding: + lea eax, [rdx + rsi] + test al, 3 + sete al + cmp rdx, 12 + setae cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rdi], rsi + mov rax, rdi + ret +.LBB5_2: + mov rax, rdx + or rax, 3 + add rax, -12 + movabs rcx, -6148914691236517205 + mov r8, rdx + mul rcx + shr rdx + mov eax, r8d + and eax, 3 + movabs rcx, 9223372036854775804 + and rcx, r8 + add rsi, rax + mov qword ptr [rdi + 8], rcx + mov qword ptr [rdi + 16], rdx + mov qword ptr [rdi], rsi + mov rax, rdi + ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..ad884432de7428 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca @@ -0,0 +1,97 @@ +Iterations: 100 +Instructions: 2900 +Total Cycles: 942 +Total uOps: 3000 + +Dispatch Width: 4 +uOps Per Cycle: 3.18 +IPC: 3.08 +Block RThroughput: 7.5 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.50 lea eax, [rdx + rsi] + 1 1 0.33 test al, 3 + 1 1 0.50 sete al + 1 1 0.33 cmp rdx, 12 + 1 1 0.50 setae cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + 1 1 0.33 mov rax, rdx + 1 1 0.33 or rax, 3 + 1 1 0.33 add rax, -12 + 1 1 0.33 movabs rcx, -6148914691236517205 + 1 1 0.33 mov r8, rdx + 2 4 1.00 mul rcx + 1 1 0.50 shr rdx + 1 1 0.33 mov eax, r8d + 1 1 0.33 and eax, 3 + 1 1 0.33 movabs rcx, 9223372036854775804 + 1 1 0.33 and rcx, r8 + 1 1 0.33 add rsi, rax + 1 1 1.00 * mov qword ptr [rdi + 8], rcx + 1 1 1.00 * mov qword ptr [rdi + 16], rdx + 1 1 1.00 * mov qword ptr [rdi], rsi + 1 1 0.33 mov rax, rdi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 8.34 8.01 4.00 8.65 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.36 0.64 - - - - lea eax, [rdx + rsi] + - - 0.32 0.36 - 0.32 - - test al, 3 + - - 0.69 - - 0.31 - - sete al + - - 0.32 0.04 - 0.64 - - cmp rdx, 12 + - - 0.68 - - 0.32 - - setae cl + - - 0.31 0.37 - 0.32 - - and cl, al + - - 0.31 0.37 - 0.32 - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi + - - 0.33 0.36 - 0.31 - - mov rax, rdi + - - - - - 1.00 - - ret + - - 0.32 0.32 - 0.36 - - mov rax, rdx + - - 0.01 0.63 - 0.36 - - or rax, 3 + - - - 0.68 - 0.32 - - add rax, -12 + - - 0.34 0.63 - 0.03 - - movabs rcx, -6148914691236517205 + - - 0.32 0.32 - 0.36 - - mov r8, rdx + - - 1.00 1.00 - - - - mul rcx + - - 0.69 - - 0.31 - - shr rdx + - - 0.37 0.32 - 0.31 - - mov eax, r8d + - - 0.35 0.33 - 0.32 - - and eax, 3 + - - 0.63 0.35 - 0.02 - - movabs rcx, 9223372036854775804 + - - 0.32 0.32 - 0.36 - - and rcx, r8 + - - 0.33 0.35 - 0.32 - - add rsi, rax + - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], rcx + - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx + - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi + - - 0.34 0.62 - 0.04 - - mov rax, rdi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs new file mode 100644 index 00000000000000..2cdadc3316d6f7 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs @@ -0,0 +1,22 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_suffix_dynamic_size( + source: &[u8], +) -> Option> { + match Ref::from_suffix(source) { + Ok((_rest, packet)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64 new file mode 100644 index 00000000000000..9dc329867dde00 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64 @@ -0,0 +1,23 @@ +bench_ref_type_from_suffix_dynamic_size: + mov rax, rdi + lea ecx, [rdx + rsi] + cmp rdx, 4 + setb dil + or dil, cl + test dil, 1 + je .LBB5_2 + xor esi, esi + mov qword ptr [rax], rsi + ret +.LBB5_2: + lea rcx, [rdx - 4] + shr rcx + mov edi, edx + and edi, 1 + movabs r8, 9223372036854775806 + and r8, rdx + add rsi, rdi + mov qword ptr [rax + 8], r8 + mov qword ptr [rax + 16], rcx + mov qword ptr [rax], rsi + ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..7639f237aa87c8 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca @@ -0,0 +1,81 @@ +Iterations: 100 +Instructions: 2100 +Total Cycles: 540 +Total uOps: 2100 + +Dispatch Width: 4 +uOps Per Cycle: 3.89 +IPC: 3.89 +Block RThroughput: 5.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea ecx, [rdx + rsi] + 1 1 0.33 cmp rdx, 4 + 1 1 0.50 setb dil + 1 1 0.33 or dil, cl + 1 1 0.33 test dil, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor esi, esi + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + 1 1 0.50 lea rcx, [rdx - 4] + 1 1 0.50 shr rcx + 1 1 0.33 mov edi, edx + 1 1 0.33 and edi, 1 + 1 1 0.33 movabs r8, 9223372036854775806 + 1 1 0.33 and r8, rdx + 1 1 0.33 add rsi, rdi + 1 1 1.00 * mov qword ptr [rax + 8], r8 + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 5.32 5.33 4.00 5.35 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.33 0.66 - 0.01 - - mov rax, rdi + - - 0.99 0.01 - - - - lea ecx, [rdx + rsi] + - - 0.01 0.98 - 0.01 - - cmp rdx, 4 + - - 0.99 - - 0.01 - - setb dil + - - - 1.00 - - - - or dil, cl + - - - 1.00 - - - - test dil, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor esi, esi + - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi + - - - - - 1.00 - - ret + - - 0.65 0.35 - - - - lea rcx, [rdx - 4] + - - 0.68 - - 0.32 - - shr rcx + - - - 0.64 - 0.36 - - mov edi, edx + - - 0.66 0.01 - 0.33 - - and edi, 1 + - - 0.34 - - 0.66 - - movabs r8, 9223372036854775806 + - - 0.32 0.68 - - - - and r8, rdx + - - 0.35 - - 0.65 - - add rsi, rdi + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], r8 + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs new file mode 100644 index 00000000000000..2ad2d2f5ee33f3 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs @@ -0,0 +1,20 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_suffix_static_size(source: &[u8]) -> Option> { + match Ref::from_suffix(source) { + Ok((_rest, packet)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64 new file mode 100644 index 00000000000000..37a3d72e7ca14b --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64 @@ -0,0 +1,18 @@ +bench_ref_type_from_suffix_static_size: + mov rax, rdi + lea ecx, [rdx + rsi] + cmp rdx, 6 + setb dil + or dil, cl + test dil, 1 + je .LBB5_2 + xor ecx, ecx + mov qword ptr [rax], rcx + ret +.LBB5_2: + lea rcx, [rsi + rdx] + add rcx, -6 + mov qword ptr [rax + 8], 6 + mov qword ptr [rax + 16], 0 + mov qword ptr [rax], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca new file mode 100644 index 00000000000000..4dfa9bda8ceb15 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca @@ -0,0 +1,71 @@ +Iterations: 100 +Instructions: 1600 +Total Cycles: 405 +Total uOps: 1600 + +Dispatch Width: 4 +uOps Per Cycle: 3.95 +IPC: 3.95 +Block RThroughput: 4.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea ecx, [rdx + rsi] + 1 1 0.33 cmp rdx, 6 + 1 1 0.50 setb dil + 1 1 0.33 or dil, cl + 1 1 0.33 test dil, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor ecx, ecx + 1 1 1.00 * mov qword ptr [rax], rcx + 1 1 1.00 U ret + 1 1 0.50 lea rcx, [rsi + rdx] + 1 1 0.33 add rcx, -6 + 1 1 1.00 * mov qword ptr [rax + 8], 6 + 1 1 1.00 * mov qword ptr [rax + 16], 0 + 1 1 1.00 * mov qword ptr [rax], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 3.66 3.66 4.00 3.68 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.66 0.33 - 0.01 - - mov rax, rdi + - - 0.33 0.67 - - - - lea ecx, [rdx + rsi] + - - 0.34 0.33 - 0.33 - - cmp rdx, 6 + - - 0.99 - - 0.01 - - setb dil + - - 0.33 0.67 - - - - or dil, cl + - - - 1.00 - - - - test dil, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor ecx, ecx + - - - - 1.00 - - 1.00 mov qword ptr [rax], rcx + - - - - - 1.00 - - ret + - - 0.34 0.66 - - - - lea rcx, [rsi + rdx] + - - 0.67 - - 0.33 - - add rcx, -6 + - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6 + - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0 + - - - - 1.00 - 1.00 - mov qword ptr [rax], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs new file mode 100644 index 00000000000000..105086dd4dc951 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs @@ -0,0 +1,23 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_suffix_with_elems_dynamic_padding( + source: &[u8], + count: usize, +) -> Option> { + match Ref::from_suffix_with_elems(source, count) { + Ok((_rest, packet)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..a52ec2c34dd99c --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_from_suffix_with_elems_dynamic_padding: + mov rax, rdi + movabs rdi, 6148914691236517201 + cmp rcx, rdi + ja .LBB5_3 + lea rdi, [rcx + 2*rcx] + or rdi, 3 + add rdi, 9 + sub rdx, rdi + setae r8b + movabs r9, 3074457345618258599 + cmp rcx, r9 + setb r9b + and r9b, r8b + cmp r9b, 1 + jne .LBB5_3 + add rsi, rdx + test sil, 3 + je .LBB5_4 +.LBB5_3: + mov qword ptr [rax], 0 + ret +.LBB5_4: + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdi + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..b8036059b98a55 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 673 +Total uOps: 2400 + +Dispatch Width: 4 +uOps Per Cycle: 3.57 +IPC: 3.57 +Block RThroughput: 6.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.33 movabs rdi, 6148914691236517201 + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 ja .LBB5_3 + 1 1 0.50 lea rdi, [rcx + 2*rcx] + 1 1 0.33 or rdi, 3 + 1 1 0.33 add rdi, 9 + 1 1 0.33 sub rdx, rdi + 1 1 0.50 setae r8b + 1 1 0.33 movabs r9, 3074457345618258599 + 1 1 0.33 cmp rcx, r9 + 1 1 0.50 setb r9b + 1 1 0.33 and r9b, r8b + 1 1 0.33 cmp r9b, 1 + 1 1 1.00 jne .LBB5_3 + 1 1 0.33 add rsi, rdx + 1 1 0.33 test sil, 3 + 1 1 1.00 je .LBB5_4 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdi + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.66 6.66 4.00 6.68 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.33 0.01 - 0.66 - - mov rax, rdi + - - 0.34 0.02 - 0.64 - - movabs rdi, 6148914691236517201 + - - 0.34 0.34 - 0.32 - - cmp rcx, rdi + - - - - - 1.00 - - ja .LBB5_3 + - - 0.66 0.34 - - - - lea rdi, [rcx + 2*rcx] + - - 0.02 0.97 - 0.01 - - or rdi, 3 + - - 0.02 0.98 - - - - add rdi, 9 + - - 0.01 0.99 - - - - sub rdx, rdi + - - 1.00 - - - - - setae r8b + - - 0.97 0.02 - 0.01 - - movabs r9, 3074457345618258599 + - - 0.98 0.01 - 0.01 - - cmp rcx, r9 + - - 0.98 - - 0.02 - - setb r9b + - - 0.01 0.98 - 0.01 - - and r9b, r8b + - - 0.02 0.98 - - - - cmp r9b, 1 + - - - - - 1.00 - - jne .LBB5_3 + - - - 1.00 - - - - add rsi, rdx + - - 0.98 0.02 - - - - test sil, 3 + - - - - - 1.00 - - je .LBB5_4 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs new file mode 100644 index 00000000000000..926f3044b4bf43 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs @@ -0,0 +1,23 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_from_suffix_with_elems_dynamic_size( + source: &[u8], + count: usize, +) -> Option> { + match Ref::from_suffix_with_elems(source, count) { + Ok((_rest, packet)) => Some(packet), + _ => None, + } +} diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64 new file mode 100644 index 00000000000000..a6c84ec3224262 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_from_suffix_with_elems_dynamic_size: + mov rax, rdi + lea rdi, [rcx + rcx] + test rcx, rcx + sets r8b + cmp rdi, -5 + setae r9b + or r9b, r8b + jne .LBB5_4 + add rdi, 4 + sub rdx, rdi + jb .LBB5_4 + movabs r8, 4611686018427387901 + cmp rcx, r8 + ja .LBB5_4 + add rsi, rdx + mov edx, esi + and edx, 1 + je .LBB5_5 +.LBB5_4: + mov qword ptr [rax], 0 + ret +.LBB5_5: + mov qword ptr [rax], rsi + mov qword ptr [rax + 8], rdi + mov qword ptr [rax + 16], rcx + ret diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..113b75b4172d28 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 673 +Total uOps: 2400 + +Dispatch Width: 4 +uOps Per Cycle: 3.57 +IPC: 3.57 +Block RThroughput: 6.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 mov rax, rdi + 1 1 0.50 lea rdi, [rcx + rcx] + 1 1 0.33 test rcx, rcx + 1 1 0.50 sets r8b + 1 1 0.33 cmp rdi, -5 + 1 1 0.50 setae r9b + 1 1 0.33 or r9b, r8b + 1 1 1.00 jne .LBB5_4 + 1 1 0.33 add rdi, 4 + 1 1 0.33 sub rdx, rdi + 1 1 1.00 jb .LBB5_4 + 1 1 0.33 movabs r8, 4611686018427387901 + 1 1 0.33 cmp rcx, r8 + 1 1 1.00 ja .LBB5_4 + 1 1 0.33 add rsi, rdx + 1 1 0.33 mov edx, esi + 1 1 0.33 and edx, 1 + 1 1 1.00 je .LBB5_5 + 1 1 1.00 * mov qword ptr [rax], 0 + 1 1 1.00 U ret + 1 1 1.00 * mov qword ptr [rax], rsi + 1 1 1.00 * mov qword ptr [rax + 8], rdi + 1 1 1.00 * mov qword ptr [rax + 16], rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.65 6.65 4.00 6.70 2.00 2.00 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - 0.99 - 0.01 - - mov rax, rdi + - - 0.01 0.99 - - - - lea rdi, [rcx + rcx] + - - 0.67 0.33 - - - - test rcx, rcx + - - 0.99 - - 0.01 - - sets r8b + - - - 1.00 - - - - cmp rdi, -5 + - - 1.00 - - - - - setae r9b + - - 1.00 - - - - - or r9b, r8b + - - - - - 1.00 - - jne .LBB5_4 + - - - 0.99 - 0.01 - - add rdi, 4 + - - - 0.33 - 0.67 - - sub rdx, rdi + - - - - - 1.00 - - jb .LBB5_4 + - - - 1.00 - - - - movabs r8, 4611686018427387901 + - - 0.66 0.34 - - - - cmp rcx, r8 + - - - - - 1.00 - - ja .LBB5_4 + - - 0.34 0.66 - - - - add rsi, rdx + - - 0.98 0.02 - - - - mov edx, esi + - - 1.00 - - - - - and edx, 1 + - - - - - 1.00 - - je .LBB5_5 + - - - - 1.00 - - 1.00 mov qword ptr [rax], 0 + - - - - - 1.00 - - ret + - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi + - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi + - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs new file mode 100644 index 00000000000000..04ac859a98a8e6 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_into_mut_dynamic_size( + source: Ref<&mut [u8], format::LocoPacket>, +) -> &mut format::LocoPacket { + Ref::into_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64 new file mode 100644 index 00000000000000..b7568f6631cee0 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_into_mut_dynamic_size: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 4611686018427387902 + cmp rdx, rdi + jae .LBB8_3 + test al, 1 + jne .LBB8_2 + lea rdi, [2*rdx + 4] + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.2] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.4] + mov rdx, rsp + mov esi, 54 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..03137756a8b52e --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 1647 +Total uOps: 2700 + +Dispatch Width: 4 +uOps Per Cycle: 1.64 +IPC: 1.46 +Block RThroughput: 6.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 4611686018427387902 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 1 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [2*rdx + 4] + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.4] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 54 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 5.15 5.74 4.00 6.11 3.98 4.02 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.72 0.28 mov rax, qword ptr [rdi] + - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8] + - - 0.83 0.16 - 0.01 - - mov esi, 1 + - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902 + - - 0.80 0.04 - 0.16 - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.13 0.74 - 0.13 - - test al, 1 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4] + - - 0.09 0.91 - - - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.29 0.13 - 0.58 - - sub rsp, 24 + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx + - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.2] + - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.4] + - - 0.09 0.68 - 0.23 - - mov rdx, rsp + - - 0.16 0.84 - - - - mov esi, 54 + - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.rs b/rust/zerocopy/benches/ref_type_into_mut_static_size.rs new file mode 100644 index 00000000000000..edab9abe0aa819 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_into_mut_static_size( + source: Ref<&mut [u8], format::LocoPacket>, +) -> &mut format::LocoPacket { + Ref::into_mut(source) +} diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64 new file mode 100644 index 00000000000000..0412b33d6cc3c2 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64 @@ -0,0 +1,3 @@ +bench_ref_type_into_mut_static_size: + mov rax, qword ptr [rdi] + ret diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca new file mode 100644 index 00000000000000..2de0e6f9791c9f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca @@ -0,0 +1,43 @@ +Iterations: 100 +Instructions: 200 +Total Cycles: 103 +Total uOps: 200 + +Dispatch Width: 4 +uOps Per Cycle: 1.94 +IPC: 1.94 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 0.50 0.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs new file mode 100644 index 00000000000000..9f5f3b618c2fff --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_padding.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_into_ref_dynamic_padding( + source: Ref<&[u8], format::LocoPacket>, +) -> &format::LocoPacket { + Ref::into_ref(source) +} diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64 new file mode 100644 index 00000000000000..ca16dbc03e8ee3 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64 @@ -0,0 +1,29 @@ +bench_ref_type_into_ref_dynamic_padding: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 3074457345618258599 + cmp rdx, rdi + jae .LBB8_3 + test al, 3 + jne .LBB8_2 + lea rdi, [rdx + 2*rdx] + or rdi, 3 + add rdi, 9 + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.2] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.4] + mov rdx, rsp + mov esi, 54 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca new file mode 100644 index 00000000000000..bf8eade92561c7 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca @@ -0,0 +1,91 @@ +Iterations: 100 +Instructions: 2600 +Total Cycles: 1812 +Total uOps: 2900 + +Dispatch Width: 4 +uOps Per Cycle: 1.60 +IPC: 1.43 +Block RThroughput: 7.3 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 3074457345618258599 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 3 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [rdx + 2*rdx] + 1 1 0.33 or rdi, 3 + 1 1 0.33 add rdi, 9 + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.4] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 54 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 6.41 6.42 4.00 6.17 3.99 4.01 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.54 0.46 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - - 0.49 0.51 mov rcx, qword ptr [rdi + 8] + - - 0.52 0.47 - 0.01 - - mov esi, 1 + - - 0.47 0.53 - - - - movabs rdi, 3074457345618258599 + - - 0.47 0.53 - - - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.54 0.44 - 0.02 - - test al, 3 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.44 0.56 - - - - lea rdi, [rdx + 2*rdx] + - - 0.48 0.52 - - - - or rdi, 3 + - - 0.48 0.48 - 0.04 - - add rdi, 9 + - - 0.51 0.39 - 0.10 - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.53 0.47 - - - - sub rsp, 24 + - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.48 0.52 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp + 16], rcx + - - 0.49 0.51 - - - - lea rdi, [rip + .Lanon.HASH.2] + - - 0.46 0.54 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.54 0.46 - - - - lea r8, [rip + .Lanon.HASH.4] + - - 0.04 0.06 - 0.90 - - mov rdx, rsp + - - 0.44 0.46 - 0.10 - - mov esi, 54 + - - - - 1.00 1.00 0.94 1.06 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs new file mode 100644 index 00000000000000..535c679852005f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_dynamic_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_into_ref_dynamic_size( + source: Ref<&[u8], format::LocoPacket>, +) -> &format::LocoPacket { + Ref::into_ref(source) +} diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64 new file mode 100644 index 00000000000000..81624de69046de --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64 @@ -0,0 +1,27 @@ +bench_ref_type_into_ref_dynamic_size: + mov rdx, qword ptr [rdi + 16] + mov rax, qword ptr [rdi] + mov rcx, qword ptr [rdi + 8] + mov esi, 1 + movabs rdi, 4611686018427387902 + cmp rdx, rdi + jae .LBB8_3 + test al, 1 + jne .LBB8_2 + lea rdi, [2*rdx + 4] + cmp rcx, rdi + jne .LBB8_3 + ret +.LBB8_2: + xor esi, esi +.LBB8_3: + sub rsp, 24 + mov qword ptr [rsp], rsi + mov qword ptr [rsp + 8], rax + mov qword ptr [rsp + 16], rcx + lea rdi, [rip + .Lanon.HASH.2] + lea rcx, [rip + .Lanon.HASH.1] + lea r8, [rip + .Lanon.HASH.4] + mov rdx, rsp + mov esi, 54 + call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca new file mode 100644 index 00000000000000..03137756a8b52e --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca @@ -0,0 +1,87 @@ +Iterations: 100 +Instructions: 2400 +Total Cycles: 1647 +Total uOps: 2700 + +Dispatch Width: 4 +uOps Per Cycle: 1.64 +IPC: 1.46 +Block RThroughput: 6.8 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rdx, qword ptr [rdi + 16] + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov rcx, qword ptr [rdi + 8] + 1 1 0.33 mov esi, 1 + 1 1 0.33 movabs rdi, 4611686018427387902 + 1 1 0.33 cmp rdx, rdi + 1 1 1.00 jae .LBB8_3 + 1 1 0.33 test al, 1 + 1 1 1.00 jne .LBB8_2 + 1 1 0.50 lea rdi, [2*rdx + 4] + 1 1 0.33 cmp rcx, rdi + 1 1 1.00 jne .LBB8_3 + 1 1 1.00 U ret + 1 0 0.25 xor esi, esi + 1 1 0.33 sub rsp, 24 + 1 1 1.00 * mov qword ptr [rsp], rsi + 1 1 1.00 * mov qword ptr [rsp + 8], rax + 1 1 1.00 * mov qword ptr [rsp + 16], rcx + 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2] + 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1] + 1 1 0.50 lea r8, [rip + .Lanon.HASH.4] + 1 1 0.33 mov rdx, rsp + 1 1 0.33 mov esi, 54 + 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 5.15 5.74 4.00 6.11 3.98 4.02 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16] + - - - - - - 0.72 0.28 mov rax, qword ptr [rdi] + - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8] + - - 0.83 0.16 - 0.01 - - mov esi, 1 + - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902 + - - 0.80 0.04 - 0.16 - - cmp rdx, rdi + - - - - - 1.00 - - jae .LBB8_3 + - - 0.13 0.74 - 0.13 - - test al, 1 + - - - - - 1.00 - - jne .LBB8_2 + - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4] + - - 0.09 0.91 - - - - cmp rcx, rdi + - - - - - 1.00 - - jne .LBB8_3 + - - - - - 1.00 - - ret + - - - - - - - - xor esi, esi + - - 0.29 0.13 - 0.58 - - sub rsp, 24 + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi + - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax + - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx + - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.2] + - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1] + - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.4] + - - 0.09 0.68 - 0.23 - - mov rdx, rsp + - - 0.16 0.84 - - - - mov esi, 54 + - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL] diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.rs b/rust/zerocopy/benches/ref_type_into_ref_static_size.rs new file mode 100644 index 00000000000000..a2bdf559a90397 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.rs @@ -0,0 +1,19 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_into_ref_static_size( + source: Ref<&[u8], format::LocoPacket>, +) -> &format::LocoPacket { + Ref::into_ref(source) +} diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64 new file mode 100644 index 00000000000000..173562c465610a --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64 @@ -0,0 +1,3 @@ +bench_ref_type_into_ref_static_size: + mov rax, qword ptr [rdi] + ret diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca new file mode 100644 index 00000000000000..2de0e6f9791c9f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca @@ -0,0 +1,43 @@ +Iterations: 100 +Instructions: 200 +Total Cycles: 103 +Total uOps: 200 + +Dispatch Width: 4 +uOps Per Cycle: 1.94 +IPC: 1.94 +Block RThroughput: 1.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - - - - 1.00 0.50 0.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_read_static_size.rs b/rust/zerocopy/benches/ref_type_read_static_size.rs new file mode 100644 index 00000000000000..da7111261b8069 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_read_static_size.rs @@ -0,0 +1,17 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_read_static_size(source: &Ref<&[u8], format::LocoPacket>) -> format::LocoPacket { + Ref::read(source) +} diff --git a/rust/zerocopy/benches/ref_type_read_static_size.x86-64 b/rust/zerocopy/benches/ref_type_read_static_size.x86-64 new file mode 100644 index 00000000000000..c5ebb527f419fb --- /dev/null +++ b/rust/zerocopy/benches/ref_type_read_static_size.x86-64 @@ -0,0 +1,7 @@ +bench_ref_type_read_static_size: + mov rax, qword ptr [rdi] + mov ecx, dword ptr [rax] + movzx eax, word ptr [rax + 4] + shl rax, 32 + or rax, rcx + ret diff --git a/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca new file mode 100644 index 00000000000000..37cfa368ce169f --- /dev/null +++ b/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca @@ -0,0 +1,51 @@ +Iterations: 100 +Instructions: 600 +Total Cycles: 169 +Total uOps: 600 + +Dispatch Width: 4 +uOps Per Cycle: 3.55 +IPC: 3.55 +Block RThroughput: 1.5 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 5 0.50 * mov ecx, dword ptr [rax] + 1 5 0.50 * movzx eax, word ptr [rax + 4] + 1 1 0.50 shl rax, 32 + 1 1 0.33 or rax, rcx + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 0.91 0.90 - 1.19 1.50 1.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - - - 0.31 0.69 mov ecx, dword ptr [rax] + - - - - - - 0.69 0.31 movzx eax, word ptr [rax + 4] + - - 0.91 - - 0.09 - - shl rax, 32 + - - - 0.90 - 0.10 - - or rax, rcx + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/ref_type_write_static_size.rs b/rust/zerocopy/benches/ref_type_write_static_size.rs new file mode 100644 index 00000000000000..6537cc4609e946 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_write_static_size.rs @@ -0,0 +1,20 @@ +// Copyright 2026 The Fuchsia Authors +// +// Licensed under a BSD-style license , Apache License, Version 2.0 +// , or the MIT +// license , at your option. +// This file may not be copied, modified, or distributed except according to +// those terms. + +use zerocopy::Ref; + +#[path = "formats/coco_static_size.rs"] +mod format; + +#[unsafe(no_mangle)] +fn bench_ref_type_write_static_size( + destination: &mut Ref<&mut [u8], format::LocoPacket>, + value: format::LocoPacket, +) { + Ref::write(destination, value) +} diff --git a/rust/zerocopy/benches/ref_type_write_static_size.x86-64 b/rust/zerocopy/benches/ref_type_write_static_size.x86-64 new file mode 100644 index 00000000000000..1b919d01409a85 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_write_static_size.x86-64 @@ -0,0 +1,6 @@ +bench_ref_type_write_static_size: + mov rax, qword ptr [rdi] + mov dword ptr [rax], esi + shr rsi, 32 + mov word ptr [rax + 4], si + ret diff --git a/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca new file mode 100644 index 00000000000000..a7089d3b133640 --- /dev/null +++ b/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca @@ -0,0 +1,49 @@ +Iterations: 100 +Instructions: 500 +Total Cycles: 208 +Total uOps: 500 + +Dispatch Width: 4 +uOps Per Cycle: 2.40 +IPC: 2.40 +Block RThroughput: 2.0 + + +Instruction Info: +[1]: #uOps +[2]: Latency +[3]: RThroughput +[4]: MayLoad +[5]: MayStore +[6]: HasSideEffects (U) + +[1] [2] [3] [4] [5] [6] Instructions: + 1 5 0.50 * mov rax, qword ptr [rdi] + 1 1 1.00 * mov dword ptr [rax], esi + 1 1 0.50 shr rsi, 32 + 1 1 1.00 * mov word ptr [rax + 4], si + 1 1 1.00 U ret + + +Resources: +[0] - SBDivider +[1] - SBFPDivider +[2] - SBPort0 +[3] - SBPort1 +[4] - SBPort4 +[5] - SBPort5 +[6.0] - SBPort23 +[6.1] - SBPort23 + + +Resource pressure per iteration: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] + - - 0.99 - 2.00 1.01 1.50 1.50 + +Resource pressure by instruction: +[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - - - - - 0.50 0.50 mov rax, qword ptr [rdi] + - - - - 1.00 - 0.33 0.67 mov dword ptr [rax], esi + - - 0.99 - - 0.01 - - shr rsi, 32 + - - - - 1.00 - 0.67 0.33 mov word ptr [rax + 4], si + - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 index 03684cb09f8faa..0a8a4f501a0750 100644 --- a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 @@ -1,9 +1,8 @@ bench_split_via_runtime_check_dynamic_padding: mov rax, rdi mov rdx, qword ptr [rsi + 16] - mov ecx, edx - and ecx, 3 - cmp ecx, 1 + lea ecx, [rdx - 1] + test cl, 3 jne .LBB5_1 mov rcx, qword ptr [rsi] mov rsi, qword ptr [rsi + 8] diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca index 5034ab0583bdb9..7140b06e884246 100644 --- a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca @@ -1,11 +1,11 @@ Iterations: 100 -Instructions: 2000 +Instructions: 1900 Total Cycles: 708 -Total uOps: 2000 +Total uOps: 1900 Dispatch Width: 4 -uOps Per Cycle: 2.82 -IPC: 2.82 +uOps Per Cycle: 2.68 +IPC: 2.68 Block RThroughput: 5.0 @@ -20,9 +20,8 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 mov rax, rdi 1 5 0.50 * mov rdx, qword ptr [rsi + 16] - 1 1 0.33 mov ecx, edx - 1 1 0.33 and ecx, 3 - 1 1 0.33 cmp ecx, 1 + 1 1 0.50 lea ecx, [rdx - 1] + 1 1 0.33 test cl, 3 1 1 1.00 jne .LBB5_1 1 5 0.50 * mov rcx, qword ptr [rsi] 1 5 0.50 * mov rsi, qword ptr [rsi + 8] @@ -53,22 +52,21 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 3.00 3.02 5.00 4.98 4.00 4.00 + - - 2.99 3.00 5.00 4.01 4.00 4.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 0.99 - 0.01 - - mov rax, rdi + - - 0.99 - - 0.01 - - mov rax, rdi - - - - - - - 1.00 mov rdx, qword ptr [rsi + 16] - - - 0.99 0.01 - - - - mov ecx, edx - - - 0.99 0.01 - - - - and ecx, 3 - - - 0.97 0.03 - - - - cmp ecx, 1 + - - - 1.00 - - - - lea ecx, [rdx - 1] + - - - 0.98 - 0.02 - - test cl, 3 - - - - - 1.00 - - jne .LBB5_1 - - - - - - 1.00 - mov rcx, qword ptr [rsi] - - - - - - 0.99 0.01 mov rsi, qword ptr [rsi + 8] - - - 0.01 0.99 - - - - lea rdi, [rdx + 2*rdx] - - - - 0.96 - 0.04 - - add rdi, rcx - - - 0.03 - - 0.97 - - add rdi, 9 - - - 0.01 0.03 - 0.96 - - sub rsi, rdx + - - 1.00 - - - - - lea rdi, [rdx + 2*rdx] + - - 0.01 0.01 - 0.98 - - add rdi, rcx + - - - 1.00 - - - - add rdi, 9 + - - 0.99 0.01 - - - - sub rsi, rdx - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rdi - - - - 1.00 - 0.01 0.99 mov qword ptr [rax + 24], rsi diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64 b/rust/zerocopy/benches/try_read_from_bytes.x86-64 index 08088a08fd8588..2135facc2655da 100644 --- a/rust/zerocopy/benches/try_read_from_bytes.x86-64 +++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64 @@ -1,23 +1,11 @@ bench_try_read_from_bytes_static_size: - mov ax, -16191 + mov eax, 49345 cmp rsi, 6 - jne .LBB5_1 - mov ecx, dword ptr [rdi] - movzx edx, cx - cmp edx, 49344 - jne .LBB5_4 - movzx eax, word ptr [rdi + 4] - shl rax, 32 - or rcx, rax - shr rcx, 16 - mov ax, -16192 -.LBB5_4: - shl rcx, 16 - movzx eax, ax - or rax, rcx - ret -.LBB5_1: - shl rcx, 16 - movzx eax, ax - or rax, rcx + jne .LBB5_3 + cmp word ptr [rdi], -16192 + jne .LBB5_3 + mov eax, dword ptr [rdi + 2] + shl rax, 16 + or rax, 49344 +.LBB5_3: ret diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca index 385e6a48025392..f5568fef5ae7b8 100644 --- a/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca +++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2000 -Total Cycles: 608 -Total uOps: 2000 +Instructions: 900 +Total Cycles: 305 +Total uOps: 1000 Dispatch Width: 4 -uOps Per Cycle: 3.29 -IPC: 3.29 -Block RThroughput: 5.0 +uOps Per Cycle: 3.28 +IPC: 2.95 +Block RThroughput: 3.0 Instruction Info: @@ -18,25 +18,14 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 mov ax, -16191 + 1 1 0.33 mov eax, 49345 1 1 0.33 cmp rsi, 6 - 1 1 1.00 jne .LBB5_1 - 1 5 0.50 * mov ecx, dword ptr [rdi] - 1 1 0.33 movzx edx, cx - 1 1 0.33 cmp edx, 49344 - 1 1 1.00 jne .LBB5_4 - 1 5 0.50 * movzx eax, word ptr [rdi + 4] - 1 1 0.50 shl rax, 32 - 1 1 0.33 or rcx, rax - 1 1 0.50 shr rcx, 16 - 1 1 0.33 mov ax, -16192 - 1 1 0.50 shl rcx, 16 - 1 1 0.33 movzx eax, ax - 1 1 0.33 or rax, rcx - 1 1 1.00 U ret - 1 1 0.50 shl rcx, 16 - 1 1 0.33 movzx eax, ax - 1 1 0.33 or rax, rcx + 1 1 1.00 jne .LBB5_3 + 2 6 0.50 * cmp word ptr [rdi], -16192 + 1 1 1.00 jne .LBB5_3 + 1 5 0.50 * mov eax, dword ptr [rdi + 2] + 1 1 0.50 shl rax, 16 + 1 1 0.33 or rax, 49344 1 1 1.00 U ret @@ -53,27 +42,16 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 5.99 5.99 - 6.02 1.00 1.00 + - - 2.50 2.49 - 3.01 1.00 1.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 0.99 - 0.01 - - mov ax, -16191 - - - - 0.01 - 0.99 - - cmp rsi, 6 - - - - - - 1.00 - - jne .LBB5_1 - - - - - - - - 1.00 mov ecx, dword ptr [rdi] - - - 0.98 - - 0.02 - - movzx edx, cx - - - 0.99 0.01 - - - - cmp edx, 49344 - - - - - - 1.00 - - jne .LBB5_4 - - - - - - - 1.00 - movzx eax, word ptr [rdi + 4] - - - 0.01 - - 0.99 - - shl rax, 32 - - - 0.02 0.98 - - - - or rcx, rax - - - 1.00 - - - - - shr rcx, 16 - - - 0.99 0.01 - - - - mov ax, -16192 - - - 1.00 - - - - - shl rcx, 16 - - - - 1.00 - - - - movzx eax, ax - - - - 1.00 - - - - or rax, rcx - - - - - - 1.00 - - ret - - - 1.00 - - - - - shl rcx, 16 - - - - 1.00 - - - - movzx eax, ax - - - - 0.99 - 0.01 - - or rax, rcx + - - 0.50 0.49 - 0.01 - - mov eax, 49345 + - - 0.49 0.51 - - - - cmp rsi, 6 + - - - - - 1.00 - - jne .LBB5_3 + - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi], -16192 + - - - - - 1.00 - - jne .LBB5_3 + - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + 2] + - - 1.00 - - - - - shl rax, 16 + - - 0.04 0.96 - - - - or rax, 49344 - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64 b/rust/zerocopy/benches/try_read_from_prefix.x86-64 index d3e1edc3ea73b4..0f820efff5f874 100644 --- a/rust/zerocopy/benches/try_read_from_prefix.x86-64 +++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64 @@ -1,16 +1,11 @@ bench_try_read_from_prefix_static_size: mov eax, 49345 cmp rsi, 6 - jb .LBB5_2 - mov eax, dword ptr [rdi] - movzx ecx, word ptr [rdi + 4] - shl rcx, 32 - or rcx, rax - movzx eax, cx - and rcx, -65536 - or rcx, 49344 - cmp eax, 49344 - mov eax, 49345 - cmove rax, rcx -.LBB5_2: + jb .LBB5_3 + cmp word ptr [rdi], -16192 + jne .LBB5_3 + mov eax, dword ptr [rdi + 2] + shl rax, 16 + or rax, 49344 +.LBB5_3: ret diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca index 40401d89e83d89..e0ba4c8320e37e 100644 --- a/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca +++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1400 -Total Cycles: 442 -Total uOps: 1500 +Instructions: 900 +Total Cycles: 305 +Total uOps: 1000 Dispatch Width: 4 -uOps Per Cycle: 3.39 -IPC: 3.17 -Block RThroughput: 3.8 +uOps Per Cycle: 3.28 +IPC: 2.95 +Block RThroughput: 3.0 Instruction Info: @@ -20,17 +20,12 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 mov eax, 49345 1 1 0.33 cmp rsi, 6 - 1 1 1.00 jb .LBB5_2 - 1 5 0.50 * mov eax, dword ptr [rdi] - 1 5 0.50 * movzx ecx, word ptr [rdi + 4] - 1 1 0.50 shl rcx, 32 - 1 1 0.33 or rcx, rax - 1 1 0.33 movzx eax, cx - 1 1 0.33 and rcx, -65536 - 1 1 0.33 or rcx, 49344 - 1 1 0.33 cmp eax, 49344 - 1 1 0.33 mov eax, 49345 - 2 2 0.67 cmove rax, rcx + 1 1 1.00 jb .LBB5_3 + 2 6 0.50 * cmp word ptr [rdi], -16192 + 1 1 1.00 jne .LBB5_3 + 1 5 0.50 * mov eax, dword ptr [rdi + 2] + 1 1 0.50 shl rax, 16 + 1 1 0.33 or rax, 49344 1 1 1.00 U ret @@ -47,21 +42,16 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 4.33 4.33 - 4.34 1.00 1.00 + - - 2.50 2.49 - 3.01 1.00 1.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.65 0.01 - 0.34 - - mov eax, 49345 - - - 0.01 0.33 - 0.66 - - cmp rsi, 6 - - - - - - 1.00 - - jb .LBB5_2 - - - - - - - - 1.00 mov eax, dword ptr [rdi] - - - - - - - 1.00 - movzx ecx, word ptr [rdi + 4] - - - 0.65 - - 0.35 - - shl rcx, 32 - - - - 0.67 - 0.33 - - or rcx, rax - - - 0.01 0.99 - - - - movzx eax, cx - - - 0.99 0.01 - - - - and rcx, -65536 - - - 0.01 0.99 - - - - or rcx, 49344 - - - 0.99 0.01 - - - - cmp eax, 49344 - - - 0.02 0.33 - 0.65 - - mov eax, 49345 - - - 1.00 0.99 - 0.01 - - cmove rax, rcx + - - 0.50 0.49 - 0.01 - - mov eax, 49345 + - - 0.49 0.51 - - - - cmp rsi, 6 + - - - - - 1.00 - - jb .LBB5_3 + - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi], -16192 + - - - - - 1.00 - - jne .LBB5_3 + - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + 2] + - - 1.00 - - - - - shl rax, 16 + - - 0.04 0.96 - - - - or rax, 49344 - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64 b/rust/zerocopy/benches/try_read_from_suffix.x86-64 index 095e326f046781..a91c074e180617 100644 --- a/rust/zerocopy/benches/try_read_from_suffix.x86-64 +++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64 @@ -1,18 +1,11 @@ bench_try_read_from_suffix_static_size: mov eax, 49345 cmp rsi, 6 - jb .LBB5_2 - mov eax, dword ptr [rdi + rsi - 6] - movzx ecx, word ptr [rdi + rsi - 2] - shl rcx, 32 - or rcx, rax - movzx edx, cx - xor eax, eax - cmp edx, 49344 - cmovne rcx, rsi - sete al - and rcx, -65536 - xor rax, 49345 - or rax, rcx -.LBB5_2: + jb .LBB5_3 + cmp word ptr [rdi + rsi - 6], -16192 + jne .LBB5_3 + mov eax, dword ptr [rdi + rsi - 4] + shl rax, 16 + or rax, 49344 +.LBB5_3: ret diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca index d3eaadbb8a813e..e42f6b93effb66 100644 --- a/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca +++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1600 -Total Cycles: 478 -Total uOps: 1700 +Instructions: 900 +Total Cycles: 305 +Total uOps: 1000 Dispatch Width: 4 -uOps Per Cycle: 3.56 -IPC: 3.35 -Block RThroughput: 4.3 +uOps Per Cycle: 3.28 +IPC: 2.95 +Block RThroughput: 3.0 Instruction Info: @@ -20,19 +20,12 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 mov eax, 49345 1 1 0.33 cmp rsi, 6 - 1 1 1.00 jb .LBB5_2 - 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 6] - 1 5 0.50 * movzx ecx, word ptr [rdi + rsi - 2] - 1 1 0.50 shl rcx, 32 - 1 1 0.33 or rcx, rax - 1 1 0.33 movzx edx, cx - 1 0 0.25 xor eax, eax - 1 1 0.33 cmp edx, 49344 - 2 2 0.67 cmovne rcx, rsi - 1 1 0.50 sete al - 1 1 0.33 and rcx, -65536 - 1 1 0.33 xor rax, 49345 - 1 1 0.33 or rax, rcx + 1 1 1.00 jb .LBB5_3 + 2 6 0.50 * cmp word ptr [rdi + rsi - 6], -16192 + 1 1 1.00 jne .LBB5_3 + 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 4] + 1 1 0.50 shl rax, 16 + 1 1 0.33 or rax, 49344 1 1 1.00 U ret @@ -49,23 +42,16 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 4.66 4.66 - 4.68 1.00 1.00 + - - 2.50 2.49 - 3.01 1.00 1.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.32 0.01 - 0.67 - - mov eax, 49345 - - - 0.62 0.02 - 0.36 - - cmp rsi, 6 - - - - - - 1.00 - - jb .LBB5_2 - - - - - - - - 1.00 mov eax, dword ptr [rdi + rsi - 6] - - - - - - - 1.00 - movzx ecx, word ptr [rdi + rsi - 2] - - - 0.37 - - 0.63 - - shl rcx, 32 - - - 0.99 0.01 - - - - or rcx, rax - - - 1.00 - - - - - movzx edx, cx - - - - - - - - - xor eax, eax - - - 0.35 0.64 - 0.01 - - cmp edx, 49344 - - - 1.00 1.00 - - - - cmovne rcx, rsi - - - - - - 1.00 - - sete al - - - 0.01 0.99 - - - - and rcx, -65536 - - - - 1.00 - - - - xor rax, 49345 - - - - 0.99 - 0.01 - - or rax, rcx + - - 0.50 0.49 - 0.01 - - mov eax, 49345 + - - 0.49 0.51 - - - - cmp rsi, 6 + - - - - - 1.00 - - jb .LBB5_3 + - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi + rsi - 6], -16192 + - - - - - 1.00 - - jne .LBB5_3 + - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + rsi - 4] + - - 1.00 - - - - - shl rax, 16 + - - 0.04 0.96 - - - - or rax, 49344 - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 index 217c5fc61796c8..5584e90541858a 100644 --- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 @@ -1,24 +1,22 @@ bench_try_ref_from_bytes_dynamic_padding: - test dil, 3 - jne .LBB5_4 - movabs rax, 9223372036854775804 - and rax, rsi - cmp rax, 9 - jb .LBB5_4 - add rax, -9 + cmp rsi, 12 + setae al + mov ecx, esi + or ecx, edi + test cl, 3 + sete cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor eax, eax + ret +.LBB5_2: + add rsi, -9 movabs rcx, -6148914691236517205 + mov rax, rsi mul rcx shr rdx - lea rax, [rdx + 2*rdx] - or rax, 3 - add rax, 9 - cmp rsi, rax - jne .LBB5_4 + xor eax, eax cmp word ptr [rdi], -16192 - je .LBB5_5 -.LBB5_4: - xor edi, edi - mov rdx, rsi -.LBB5_5: - mov rax, rdi + cmove rax, rdi ret diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca index 95b993c7e0a8a9..ccd610c7ef41f6 100644 --- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca @@ -1,10 +1,10 @@ Iterations: 100 -Instructions: 2100 -Total Cycles: 709 +Instructions: 2000 +Total Cycles: 675 Total uOps: 2300 Dispatch Width: 4 -uOps Per Cycle: 3.24 +uOps Per Cycle: 3.41 IPC: 2.96 Block RThroughput: 5.8 @@ -18,26 +18,25 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 test dil, 3 - 1 1 1.00 jne .LBB5_4 - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rax, rsi - 1 1 0.33 cmp rax, 9 - 1 1 1.00 jb .LBB5_4 - 1 1 0.33 add rax, -9 + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setae al + 1 1 0.33 mov ecx, esi + 1 1 0.33 or ecx, edi + 1 1 0.33 test cl, 3 + 1 1 0.50 sete cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor eax, eax + 1 1 1.00 U ret + 1 1 0.33 add rsi, -9 1 1 0.33 movabs rcx, -6148914691236517205 + 1 1 0.33 mov rax, rsi 2 4 1.00 mul rcx 1 1 0.50 shr rdx - 1 1 0.50 lea rax, [rdx + 2*rdx] - 1 1 0.33 or rax, 3 - 1 1 0.33 add rax, 9 - 1 1 0.33 cmp rsi, rax - 1 1 1.00 jne .LBB5_4 + 1 0 0.25 xor eax, eax 2 6 0.50 * cmp word ptr [rdi], -16192 - 1 1 1.00 je .LBB5_5 - 1 0 0.25 xor edi, edi - 1 1 0.33 mov rdx, rsi - 1 1 0.33 mov rax, rdi + 2 2 0.67 cmove rax, rdi 1 1 1.00 U ret @@ -54,28 +53,27 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.98 6.99 - 7.03 0.50 0.50 + - - 6.67 6.66 - 6.67 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.48 0.51 - 0.01 - - test dil, 3 - - - - - - 1.00 - - jne .LBB5_4 - - - 0.51 0.49 - - - - movabs rax, 9223372036854775804 - - - 0.01 0.99 - - - - and rax, rsi - - - 0.51 0.49 - - - - cmp rax, 9 - - - - - - 1.00 - - jb .LBB5_4 - - - 0.98 - - 0.02 - - add rax, -9 - - - 0.98 0.02 - - - - movabs rcx, -6148914691236517205 + - - 0.33 0.66 - 0.01 - - cmp rsi, 12 + - - 1.00 - - - - - setae al + - - 0.66 0.34 - - - - mov ecx, esi + - - - 0.99 - 0.01 - - or ecx, edi + - - - 0.67 - 0.33 - - test cl, 3 + - - 0.67 - - 0.33 - - sete cl + - - 0.33 0.34 - 0.33 - - and cl, al + - - 0.33 - - 0.67 - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor eax, eax + - - - - - 1.00 - - ret + - - 0.67 0.33 - - - - add rsi, -9 + - - 0.99 0.01 - - - - movabs rcx, -6148914691236517205 + - - - 0.99 - 0.01 - - mov rax, rsi - - 1.00 1.00 - - - - mul rcx - - - 0.99 - - 0.01 - - shr rdx - - - - 1.00 - - - - lea rax, [rdx + 2*rdx] - - - - 0.51 - 0.49 - - or rax, 3 - - - 0.01 0.49 - 0.50 - - add rax, 9 - - - - 0.02 - 0.98 - - cmp rsi, rax - - - - - - 1.00 - - jne .LBB5_4 - - - 0.51 0.49 - - 0.50 0.50 cmp word ptr [rdi], -16192 - - - - - - 1.00 - - je .LBB5_5 - - - - - - - - - xor edi, edi - - - 0.50 0.50 - - - - mov rdx, rsi - - - 0.50 0.48 - 0.02 - - mov rax, rdi + - - 0.01 - - 0.99 - - shr rdx + - - - - - - - - xor eax, eax + - - 0.67 0.33 - - 0.50 0.50 cmp word ptr [rdi], -16192 + - - 0.01 1.00 - 0.99 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 index cf67afd31ce07b..39b12e14eec120 100644 --- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 @@ -1,22 +1,21 @@ bench_try_ref_from_bytes_dynamic_size: - mov rdx, rsi - mov rax, rdi cmp rsi, 4 - setb cl - or cl, al + setae al + mov ecx, esi + or ecx, edi test cl, 1 - jne .LBB5_4 - lea rcx, [rdx - 4] - mov rsi, rcx - and rsi, -2 - add rsi, 4 - cmp rdx, rsi - jne .LBB5_4 - cmp word ptr [rax], -16192 - jne .LBB5_4 - shr rcx - mov rdx, rcx + sete cl + and cl, al + cmp cl, 1 + je .LBB5_2 + xor eax, eax ret -.LBB5_4: +.LBB5_2: + lea rdx, [rsi - 4] + shr rdx + movzx ecx, word ptr [rdi] xor eax, eax + cmp ecx, 49344 + cmovne rdx, rsi + cmove rax, rdi ret diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca index ecd7a18f6d6d45..eecf3a1554cb9e 100644 --- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca @@ -1,10 +1,10 @@ Iterations: 100 -Instructions: 2000 -Total Cycles: 639 +Instructions: 1900 +Total Cycles: 607 Total uOps: 2100 Dispatch Width: 4 -uOps Per Cycle: 3.29 +uOps Per Cycle: 3.46 IPC: 3.13 Block RThroughput: 5.3 @@ -18,25 +18,24 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.33 mov rdx, rsi - 1 1 0.33 mov rax, rdi 1 1 0.33 cmp rsi, 4 - 1 1 0.50 setb cl - 1 1 0.33 or cl, al + 1 1 0.50 setae al + 1 1 0.33 mov ecx, esi + 1 1 0.33 or ecx, edi 1 1 0.33 test cl, 1 - 1 1 1.00 jne .LBB5_4 - 1 1 0.50 lea rcx, [rdx - 4] - 1 1 0.33 mov rsi, rcx - 1 1 0.33 and rsi, -2 - 1 1 0.33 add rsi, 4 - 1 1 0.33 cmp rdx, rsi - 1 1 1.00 jne .LBB5_4 - 2 6 0.50 * cmp word ptr [rax], -16192 - 1 1 1.00 jne .LBB5_4 - 1 1 0.50 shr rcx - 1 1 0.33 mov rdx, rcx + 1 1 0.50 sete cl + 1 1 0.33 and cl, al + 1 1 0.33 cmp cl, 1 + 1 1 1.00 je .LBB5_2 + 1 0 0.25 xor eax, eax 1 1 1.00 U ret + 1 1 0.50 lea rdx, [rsi - 4] + 1 1 0.50 shr rdx + 1 5 0.50 * movzx ecx, word ptr [rdi] 1 0 0.25 xor eax, eax + 1 1 0.33 cmp ecx, 49344 + 2 2 0.67 cmovne rdx, rsi + 2 2 0.67 cmove rax, rdi 1 1 1.00 U ret @@ -53,27 +52,26 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.32 6.32 - 6.36 0.50 0.50 + - - 5.99 5.99 - 6.02 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.33 0.66 - 0.01 - - mov rdx, rsi - - - 0.66 0.34 - - - - mov rax, rdi - - - 0.34 0.66 - - - - cmp rsi, 4 - - - 0.99 - - 0.01 - - setb cl - - - 0.01 0.99 - - - - or cl, al - - - - 1.00 - - - - test cl, 1 - - - - - - 1.00 - - jne .LBB5_4 - - - 0.66 0.34 - - - - lea rcx, [rdx - 4] - - - 0.33 0.66 - 0.01 - - mov rsi, rcx - - - 1.00 - - - - - and rsi, -2 - - - 0.66 0.34 - - - - add rsi, 4 - - - - 1.00 - - - - cmp rdx, rsi - - - - - - 1.00 - - jne .LBB5_4 - - - - - - 1.00 0.50 0.50 cmp word ptr [rax], -16192 - - - - - - 1.00 - - jne .LBB5_4 - - - 0.67 - - 0.33 - - shr rcx - - - 0.67 0.33 - - - - mov rdx, rcx + - - 0.02 0.97 - 0.01 - - cmp rsi, 4 + - - 1.00 - - - - - setae al + - - - 1.00 - - - - mov ecx, esi + - - 0.93 0.05 - 0.02 - - or ecx, edi + - - 0.94 0.06 - - - - test cl, 1 + - - 0.99 - - 0.01 - - sete cl + - - 0.98 0.02 - - - - and cl, al + - - 0.98 0.02 - - - - cmp cl, 1 + - - - - - 1.00 - - je .LBB5_2 + - - - - - - - - xor eax, eax - - - - - 1.00 - - ret + - - 0.01 0.99 - - - - lea rdx, [rsi - 4] + - - - - - 1.00 - - shr rdx + - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi] - - - - - - - - xor eax, eax + - - 0.02 0.96 - 0.02 - - cmp ecx, 49344 + - - 0.05 0.97 - 0.98 - - cmovne rdx, rsi + - - 0.07 0.95 - 0.98 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 index d832cb7ecf7ff4..ff64bb6595913b 100644 --- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 @@ -1,19 +1,18 @@ bench_try_ref_from_prefix_dynamic_padding: + test dil, 3 + sete al + cmp rsi, 12 + setae cl + test al, cl + jne .LBB5_2 xor edx, edx - mov eax, 0 test dil, 3 - je .LBB5_1 - ret -.LBB5_1: - movabs rax, 9223372036854775804 - and rsi, rax - cmp rsi, 9 - jae .LBB5_3 - mov edx, 1 + sete dl xor eax, eax ret -.LBB5_3: - add rsi, -9 +.LBB5_2: + or rsi, 3 + add rsi, -12 movabs rcx, -6148914691236517205 mov rax, rsi mul rcx diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca index 482112a39b3385..29225404a28f0d 100644 --- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca @@ -18,19 +18,19 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: + 1 1 0.33 test dil, 3 + 1 1 0.50 sete al + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setae cl + 1 1 0.33 test al, cl + 1 1 1.00 jne .LBB5_2 1 0 0.25 xor edx, edx - 1 1 0.33 mov eax, 0 1 1 0.33 test dil, 3 - 1 1 1.00 je .LBB5_1 - 1 1 1.00 U ret - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rsi, rax - 1 1 0.33 cmp rsi, 9 - 1 1 1.00 jae .LBB5_3 - 1 1 0.33 mov edx, 1 + 1 1 0.50 sete dl 1 0 0.25 xor eax, eax 1 1 1.00 U ret - 1 1 0.33 add rsi, -9 + 1 1 0.33 or rsi, 3 + 1 1 0.33 add rsi, -12 1 1 0.33 movabs rcx, -6148914691236517205 1 1 0.33 mov rax, rsi 2 4 1.00 mul rcx @@ -63,29 +63,29 @@ Resource pressure per iteration: Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: + - - 0.01 0.98 - 0.01 - - test dil, 3 + - - 0.98 - - 0.02 - - sete al + - - - 1.00 - - - - cmp rsi, 12 + - - - - - 1.00 - - setae cl + - - 0.01 0.03 - 0.96 - - test al, cl + - - - - - 1.00 - - jne .LBB5_2 - - - - - - - - xor edx, edx - - - 0.32 0.34 - 0.34 - - mov eax, 0 - - - 0.34 0.33 - 0.33 - - test dil, 3 - - - - - - 1.00 - - je .LBB5_1 - - - - - - 1.00 - - ret - - - 0.35 0.65 - - - - movabs rax, 9223372036854775804 - - - 0.96 0.03 - 0.01 - - and rsi, rax - - - 0.01 0.97 - 0.02 - - cmp rsi, 9 - - - - - - 1.00 - - jae .LBB5_3 - - - 0.67 0.01 - 0.32 - - mov edx, 1 + - - - 1.00 - - - - test dil, 3 + - - 0.99 - - 0.01 - - sete dl - - - - - - - - xor eax, eax - - - - - 1.00 - - ret - - - 0.02 0.34 - 0.64 - - add rsi, -9 - - - 0.33 0.66 - 0.01 - - movabs rcx, -6148914691236517205 - - - 0.66 0.34 - - - - mov rax, rsi + - - - 1.00 - - - - or rsi, 3 + - - 0.01 0.99 - - - - add rsi, -12 + - - 0.01 0.99 - - - - movabs rcx, -6148914691236517205 + - - 0.34 0.66 - - - - mov rax, rsi - - 1.00 1.00 - - - - mul rcx - - - 0.01 0.99 - - - - mov rax, rdx - - - 0.99 - - 0.01 - - shr rax + - - 0.99 - - 0.01 - - mov rax, rdx + - - 1.00 - - - - - shr rax - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi] - - - 0.33 0.03 - 0.64 - - cmp cx, -16192 - - - 0.01 0.31 - 0.68 - - mov edx, 2 - - - 1.00 1.00 - - - - cmove rdx, rax + - - 0.34 0.32 - 0.34 - - cmp cx, -16192 + - - 0.33 0.01 - 0.66 - - mov edx, 2 + - - 0.99 0.01 - 1.00 - - cmove rdx, rax - - - - - - - - xor eax, eax - - 0.33 0.33 - 0.34 - - cmp ecx, 49344 - - - 1.00 1.00 - - - - cmove rax, rdi + - - 1.00 0.01 - 0.99 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 index be7f34b9f8f5c8..de31f571f3c949 100644 --- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 @@ -1,14 +1,15 @@ bench_try_ref_from_prefix_dynamic_size: - xor edx, edx - mov eax, 0 - test dil, 1 - jne .LBB5_4 cmp rsi, 4 - jae .LBB5_3 - mov edx, 1 + setb al + or al, dil + test al, 1 + je .LBB5_2 + not edi + and edi, 1 xor eax, eax + mov rdx, rdi ret -.LBB5_3: +.LBB5_2: add rsi, -4 shr rsi movzx ecx, word ptr [rdi] @@ -18,5 +19,4 @@ bench_try_ref_from_prefix_dynamic_size: xor eax, eax cmp cx, -16192 cmove rax, rdi -.LBB5_4: ret diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca index 11706defe11e85..cd0919d04c0842 100644 --- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1900 -Total Cycles: 573 -Total uOps: 2100 +Instructions: 2000 +Total Cycles: 643 +Total uOps: 2200 Dispatch Width: 4 -uOps Per Cycle: 3.66 -IPC: 3.32 -Block RThroughput: 5.3 +uOps Per Cycle: 3.42 +IPC: 3.11 +Block RThroughput: 5.5 Instruction Info: @@ -18,14 +18,15 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 0 0.25 xor edx, edx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test dil, 1 - 1 1 1.00 jne .LBB5_4 1 1 0.33 cmp rsi, 4 - 1 1 1.00 jae .LBB5_3 - 1 1 0.33 mov edx, 1 + 1 1 0.50 setb al + 1 1 0.33 or al, dil + 1 1 0.33 test al, 1 + 1 1 1.00 je .LBB5_2 + 1 1 0.33 not edi + 1 1 0.33 and edi, 1 1 0 0.25 xor eax, eax + 1 1 0.33 mov rdx, rdi 1 1 1.00 U ret 1 1 0.33 add rsi, -4 1 1 0.50 shr rsi @@ -52,26 +53,27 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 5.66 5.67 - 5.67 0.50 0.50 + - - 6.33 6.33 - 6.34 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - - - - - - xor edx, edx - - - 0.30 0.37 - 0.33 - - mov eax, 0 - - - 0.35 0.32 - 0.33 - - test dil, 1 - - - - - - 1.00 - - jne .LBB5_4 - - - 0.32 0.33 - 0.35 - - cmp rsi, 4 - - - - - - 1.00 - - jae .LBB5_3 - - - 0.33 0.35 - 0.32 - - mov edx, 1 + - - 0.67 0.32 - 0.01 - - cmp rsi, 4 + - - 0.99 - - 0.01 - - setb al + - - 0.19 0.48 - 0.33 - - or al, dil + - - 0.50 0.16 - 0.34 - - test al, 1 + - - - - - 1.00 - - je .LBB5_2 + - - 0.50 0.50 - - - - not edi + - - 0.49 0.34 - 0.17 - - and edi, 1 - - - - - - - - xor eax, eax + - - 0.16 0.84 - - - - mov rdx, rdi - - - - - 1.00 - - ret - - - 0.34 0.64 - 0.02 - - add rsi, -4 - - - 1.00 - - - - - shr rsi + - - 0.68 0.32 - - - - add rsi, -4 + - - 0.85 - - 0.15 - - shr rsi - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi] - - - 0.60 0.40 - - - - cmp ecx, 49344 - - - 0.05 0.95 - - - - mov edx, 2 - - - 1.00 1.00 - - - - cmove rdx, rsi + - - - 0.50 - 0.50 - - cmp ecx, 49344 + - - 0.63 0.37 - - - - mov edx, 2 + - - 0.17 1.00 - 0.83 - - cmove rdx, rsi - - - - - - - - xor eax, eax - - - 0.37 0.31 - 0.32 - - cmp cx, -16192 - - - 1.00 1.00 - - - - cmove rax, rdi + - - 0.50 0.50 - - - - cmp cx, -16192 + - - - 1.00 - 1.00 - - cmove rax, rdi - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 index b3e924442865f4..cee9a3179097ce 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 @@ -1,26 +1,23 @@ bench_try_ref_from_suffix_dynamic_padding: lea eax, [rsi + rdi] test al, 3 - jne .LBB5_1 - movabs rax, 9223372036854775804 - and rax, rsi - cmp rax, 9 - jae .LBB5_3 -.LBB5_1: + setne al + cmp rsi, 12 + setb cl + or cl, al + je .LBB5_2 xor eax, eax ret -.LBB5_3: - add rax, -9 +.LBB5_2: + mov rax, rsi + or rax, 3 + add rax, -12 movabs rcx, -6148914691236517205 mul rcx shr rdx - lea rcx, [rdx + 2*rdx] - sub rsi, rcx - or rcx, -4 - add rsi, rdi - lea rdi, [rcx + rsi] - add rdi, -8 + and esi, 3 + lea rcx, [rdi + rsi] xor eax, eax - cmp word ptr [rcx + rsi - 8], -16192 - cmove rax, rdi + cmp word ptr [rdi + rsi], -16192 + cmove rax, rcx ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca index d56ae56d854a68..92df4484b82397 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2300 -Total Cycles: 791 -Total uOps: 2600 +Instructions: 2100 +Total Cycles: 710 +Total uOps: 2400 Dispatch Width: 4 -uOps Per Cycle: 3.29 -IPC: 2.91 -Block RThroughput: 6.5 +uOps Per Cycle: 3.38 +IPC: 2.96 +Block RThroughput: 6.0 Instruction Info: @@ -20,26 +20,24 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.50 lea eax, [rsi + rdi] 1 1 0.33 test al, 3 - 1 1 1.00 jne .LBB5_1 - 1 1 0.33 movabs rax, 9223372036854775804 - 1 1 0.33 and rax, rsi - 1 1 0.33 cmp rax, 9 - 1 1 1.00 jae .LBB5_3 + 1 1 0.50 setne al + 1 1 0.33 cmp rsi, 12 + 1 1 0.50 setb cl + 1 1 0.33 or cl, al + 1 1 1.00 je .LBB5_2 1 0 0.25 xor eax, eax 1 1 1.00 U ret - 1 1 0.33 add rax, -9 + 1 1 0.33 mov rax, rsi + 1 1 0.33 or rax, 3 + 1 1 0.33 add rax, -12 1 1 0.33 movabs rcx, -6148914691236517205 2 4 1.00 mul rcx 1 1 0.50 shr rdx - 1 1 0.50 lea rcx, [rdx + 2*rdx] - 1 1 0.33 sub rsi, rcx - 1 1 0.33 or rcx, -4 - 1 1 0.33 add rsi, rdi - 1 1 0.50 lea rdi, [rcx + rsi] - 1 1 0.33 add rdi, -8 + 1 1 0.33 and esi, 3 + 1 1 0.50 lea rcx, [rdi + rsi] 1 0 0.25 xor eax, eax - 2 6 0.50 * cmp word ptr [rcx + rsi - 8], -16192 - 2 2 0.67 cmove rax, rdi + 2 6 0.50 * cmp word ptr [rdi + rsi], -16192 + 2 2 0.67 cmove rax, rcx 1 1 1.00 U ret @@ -56,30 +54,28 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 7.70 7.58 - 7.72 0.50 0.50 + - - 7.00 7.00 - 7.00 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.26 0.74 - - - - lea eax, [rsi + rdi] - - - 0.19 0.28 - 0.53 - - test al, 3 - - - - - - 1.00 - - jne .LBB5_1 - - - 0.93 0.06 - 0.01 - - movabs rax, 9223372036854775804 - - - 0.81 0.14 - 0.05 - - and rax, rsi - - - 0.55 0.43 - 0.02 - - cmp rax, 9 - - - - - - 1.00 - - jae .LBB5_3 + - - 0.99 0.01 - - - - lea eax, [rsi + rdi] + - - 0.01 0.99 - - - - test al, 3 + - - 0.98 - - 0.02 - - setne al + - - 0.99 - - 0.01 - - cmp rsi, 12 + - - 0.02 - - 0.98 - - setb cl + - - - 0.03 - 0.97 - - or cl, al + - - - - - 1.00 - - je .LBB5_2 - - - - - - - - xor eax, eax - - - - - 1.00 - - ret - - - 0.42 0.56 - 0.02 - - add rax, -9 - - - 0.67 0.30 - 0.03 - - movabs rcx, -6148914691236517205 + - - - 1.00 - - - - mov rax, rsi + - - 0.03 0.97 - - - - or rax, 3 + - - - 1.00 - - - - add rax, -12 + - - 0.99 - - 0.01 - - movabs rcx, -6148914691236517205 - - 1.00 1.00 - - - - mul rcx - - - 0.71 - - 0.29 - - shr rdx - - - 0.32 0.68 - - - - lea rcx, [rdx + 2*rdx] - - - 0.57 0.04 - 0.39 - - sub rsi, rcx - - - 0.28 0.67 - 0.05 - - or rcx, -4 - - - 0.29 0.29 - 0.42 - - add rsi, rdi - - - 0.02 0.98 - - - - lea rdi, [rcx + rsi] - - - 0.02 0.41 - 0.57 - - add rdi, -8 + - - 0.01 - - 0.99 - - shr rdx + - - 1.00 - - - - - and esi, 3 + - - 0.97 0.03 - - - - lea rcx, [rdi + rsi] - - - - - - - - xor eax, eax - - - 0.57 0.01 - 0.42 0.50 0.50 cmp word ptr [rcx + rsi - 8], -16192 - - - 0.09 0.99 - 0.92 - - cmove rax, rdi + - - - 0.97 - 0.03 0.50 0.50 cmp word ptr [rdi + rsi], -16192 + - - 0.01 1.00 - 0.99 - - cmove rax, rcx - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 index c7530d8b681599..ca483dee63ba66 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 @@ -1,32 +1,23 @@ bench_try_ref_from_suffix_with_elems_dynamic_padding: movabs rax, 3074457345618258598 cmp rdx, rax - ja .LBB5_1 - lea r8d, [rsi + rdi] - xor ecx, ecx - mov eax, 0 - test r8b, 3 - je .LBB5_3 - mov rdx, rcx - ret -.LBB5_3: + ja .LBB5_3 lea rax, [rdx + 2*rdx] or rax, 3 add rax, 9 + lea ecx, [rsi + rdi] + test cl, 3 + setne cl sub rsi, rax - jae .LBB5_4 -.LBB5_1: + setb al + or al, cl + je .LBB5_4 +.LBB5_3: xor eax, eax - mov edx, 1 ret .LBB5_4: - lea r8, [rdi + rsi] - movzx esi, word ptr [rdi + rsi] - cmp si, -16192 - mov ecx, 2 - cmove rcx, rdx + lea rcx, [rdi + rsi] xor eax, eax - cmp esi, 49344 - cmove rax, r8 - mov rdx, rcx + cmp word ptr [rdi + rsi], -16192 + cmove rax, rcx ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca index be736c00c25060..c37a42c160f7c9 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2800 -Total Cycles: 878 -Total uOps: 3000 +Instructions: 2000 +Total Cycles: 643 +Total uOps: 2200 Dispatch Width: 4 uOps Per Cycle: 3.42 -IPC: 3.19 -Block RThroughput: 7.5 +IPC: 3.11 +Block RThroughput: 5.5 Instruction Info: @@ -20,31 +20,23 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 movabs rax, 3074457345618258598 1 1 0.33 cmp rdx, rax - 1 1 1.00 ja .LBB5_1 - 1 1 0.50 lea r8d, [rsi + rdi] - 1 0 0.25 xor ecx, ecx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test r8b, 3 - 1 1 1.00 je .LBB5_3 - 1 1 0.33 mov rdx, rcx - 1 1 1.00 U ret + 1 1 1.00 ja .LBB5_3 1 1 0.50 lea rax, [rdx + 2*rdx] 1 1 0.33 or rax, 3 1 1 0.33 add rax, 9 + 1 1 0.50 lea ecx, [rsi + rdi] + 1 1 0.33 test cl, 3 + 1 1 0.50 setne cl 1 1 0.33 sub rsi, rax - 1 1 1.00 jae .LBB5_4 + 1 1 0.50 setb al + 1 1 0.33 or al, cl + 1 1 1.00 je .LBB5_4 1 0 0.25 xor eax, eax - 1 1 0.33 mov edx, 1 1 1 1.00 U ret - 1 1 0.50 lea r8, [rdi + rsi] - 1 5 0.50 * movzx esi, word ptr [rdi + rsi] - 1 1 0.33 cmp si, -16192 - 1 1 0.33 mov ecx, 2 - 2 2 0.67 cmove rcx, rdx + 1 1 0.50 lea rcx, [rdi + rsi] 1 0 0.25 xor eax, eax - 1 1 0.33 cmp esi, 49344 - 2 2 0.67 cmove rax, r8 - 1 1 0.33 mov rdx, rcx + 2 6 0.50 * cmp word ptr [rdi + rsi], -16192 + 2 2 0.67 cmove rax, rcx 1 1 1.00 U ret @@ -61,35 +53,27 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 8.65 8.65 - 8.70 0.50 0.50 + - - 6.32 6.33 - 6.35 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.67 0.30 - 0.03 - - movabs rax, 3074457345618258598 - - - 0.01 0.99 - - - - cmp rdx, rax - - - - - - 1.00 - - ja .LBB5_1 - - - 0.99 0.01 - - - - lea r8d, [rsi + rdi] - - - - - - - - - xor ecx, ecx - - - 0.35 0.62 - 0.03 - - mov eax, 0 - - - 0.99 0.01 - - - - test r8b, 3 - - - - - - 1.00 - - je .LBB5_3 - - - 0.68 0.30 - 0.02 - - mov rdx, rcx - - - - - - 1.00 - - ret - - - 0.07 0.93 - - - - lea rax, [rdx + 2*rdx] - - - 0.06 0.35 - 0.59 - - or rax, 3 - - - 0.02 0.07 - 0.91 - - add rax, 9 - - - 0.01 0.04 - 0.95 - - sub rsi, rax - - - - - - 1.00 - - jae .LBB5_4 + - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598 + - - 0.02 0.98 - - - - cmp rdx, rax + - - - - - 1.00 - - ja .LBB5_3 + - - 0.66 0.34 - - - - lea rax, [rdx + 2*rdx] + - - 0.99 - - 0.01 - - or rax, 3 + - - 0.99 0.01 - - - - add rax, 9 + - - 0.01 0.99 - - - - lea ecx, [rsi + rdi] + - - - 0.99 - 0.01 - - test cl, 3 + - - 0.02 - - 0.98 - - setne cl + - - 0.97 0.02 - 0.01 - - sub rsi, rax + - - 0.99 - - 0.01 - - setb al + - - 0.32 0.67 - 0.01 - - or al, cl + - - - - - 1.00 - - je .LBB5_4 - - - - - - - - xor eax, eax - - - 0.92 0.01 - 0.07 - - mov edx, 1 - - - - - 1.00 - - ret - - - - 1.00 - - - - lea r8, [rdi + rsi] - - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi] - - - 0.01 0.99 - - - - cmp si, -16192 - - - 0.88 0.04 - 0.08 - - mov ecx, 2 - - - 1.00 0.99 - 0.01 - - cmove rcx, rdx + - - - 1.00 - - - - lea rcx, [rdi + rsi] - - - - - - - - xor eax, eax - - - 0.99 0.01 - - - - cmp esi, 49344 - - - 1.00 1.00 - - - - cmove rax, r8 - - - - 0.99 - 0.01 - - mov rdx, rcx + - - 0.35 0.32 - 0.33 0.50 0.50 cmp word ptr [rdi + rsi], -16192 + - - 1.00 0.02 - 0.98 - - cmove rax, rcx - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 index 952eb12de8d609..748eb1166e1321 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 @@ -1,28 +1,20 @@ bench_try_ref_from_suffix_with_elems_dynamic_size: movabs rax, 4611686018427387901 cmp rdx, rax - ja .LBB5_1 - lea r8d, [rsi + rdi] - xor ecx, ecx - mov eax, 0 - test r8b, 1 - jne .LBB5_5 + ja .LBB5_3 lea rax, [2*rdx + 4] + lea ecx, [rsi + rdi] sub rsi, rax - jae .LBB5_4 -.LBB5_1: + setb al + or al, cl + test al, 1 + je .LBB5_4 +.LBB5_3: xor eax, eax - mov edx, 1 ret .LBB5_4: - lea r8, [rdi + rsi] - movzx esi, word ptr [rdi + rsi] - cmp si, -16192 - mov ecx, 2 - cmove rcx, rdx + lea rcx, [rdi + rsi] xor eax, eax - cmp esi, 49344 - cmove rax, r8 -.LBB5_5: - mov rdx, rcx + cmp word ptr [rdi + rsi], -16192 + cmove rax, rcx ret diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca index d4f78f67a252dd..53ff0619269654 100644 --- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2400 -Total Cycles: 1107 -Total uOps: 2600 +Instructions: 1700 +Total Cycles: 544 +Total uOps: 1900 Dispatch Width: 4 -uOps Per Cycle: 2.35 -IPC: 2.17 -Block RThroughput: 6.5 +uOps Per Cycle: 3.49 +IPC: 3.13 +Block RThroughput: 4.8 Instruction Info: @@ -20,27 +20,20 @@ Instruction Info: [1] [2] [3] [4] [5] [6] Instructions: 1 1 0.33 movabs rax, 4611686018427387901 1 1 0.33 cmp rdx, rax - 1 1 1.00 ja .LBB5_1 - 1 1 0.50 lea r8d, [rsi + rdi] - 1 0 0.25 xor ecx, ecx - 1 1 0.33 mov eax, 0 - 1 1 0.33 test r8b, 1 - 1 1 1.00 jne .LBB5_5 + 1 1 1.00 ja .LBB5_3 1 1 0.50 lea rax, [2*rdx + 4] + 1 1 0.50 lea ecx, [rsi + rdi] 1 1 0.33 sub rsi, rax - 1 1 1.00 jae .LBB5_4 + 1 1 0.50 setb al + 1 1 0.33 or al, cl + 1 1 0.33 test al, 1 + 1 1 1.00 je .LBB5_4 1 0 0.25 xor eax, eax - 1 1 0.33 mov edx, 1 1 1 1.00 U ret - 1 1 0.50 lea r8, [rdi + rsi] - 1 5 0.50 * movzx esi, word ptr [rdi + rsi] - 1 1 0.33 cmp si, -16192 - 1 1 0.33 mov ecx, 2 - 2 2 0.67 cmove rcx, rdx + 1 1 0.50 lea rcx, [rdi + rsi] 1 0 0.25 xor eax, eax - 1 1 0.33 cmp esi, 49344 - 2 2 0.67 cmove rax, r8 - 1 1 0.33 mov rdx, rcx + 2 6 0.50 * cmp word ptr [rdi + rsi], -16192 + 2 2 0.67 cmove rax, rcx 1 1 1.00 U ret @@ -57,31 +50,24 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 6.99 7.00 - 8.01 0.50 0.50 + - - 5.33 5.33 - 5.34 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.02 0.95 - 0.03 - - movabs rax, 4611686018427387901 - - - 0.93 0.04 - 0.03 - - cmp rdx, rax - - - - - - 1.00 - - ja .LBB5_1 - - - 0.96 0.04 - - - - lea r8d, [rsi + rdi] - - - - - - - - - xor ecx, ecx - - - 0.95 0.02 - 0.03 - - mov eax, 0 - - - 0.95 0.05 - - - - test r8b, 1 - - - - - - 1.00 - - jne .LBB5_5 - - - 0.06 0.94 - - - - lea rax, [2*rdx + 4] - - - 0.93 0.07 - - - - sub rsi, rax - - - - - - 1.00 - - jae .LBB5_4 + - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901 + - - 0.02 0.98 - - - - cmp rdx, rax + - - - - - 1.00 - - ja .LBB5_3 + - - 0.98 0.02 - - - - lea rax, [2*rdx + 4] + - - 0.66 0.34 - - - - lea ecx, [rsi + rdi] + - - 0.33 0.65 - 0.02 - - sub rsi, rax + - - 0.99 - - 0.01 - - setb al + - - 0.01 0.36 - 0.63 - - or al, cl + - - 0.01 0.96 - 0.03 - - test al, 1 + - - - - - 1.00 - - je .LBB5_4 - - - - - - - - xor eax, eax - - - 0.03 0.95 - 0.02 - - mov edx, 1 - - - - - 1.00 - - ret - - - 0.97 0.03 - - - - lea r8, [rdi + rsi] - - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi] - - - 0.03 0.97 - - - - cmp si, -16192 - - - 0.05 0.94 - 0.01 - - mov ecx, 2 - - - 0.06 0.98 - 0.96 - - cmove rcx, rdx + - - 0.32 0.68 - - - - lea rcx, [rdi + rsi] - - - - - - - - xor eax, eax - - - 0.97 0.03 - - - - cmp esi, 49344 - - - 0.06 0.96 - 0.98 - - cmove rax, r8 - - - 0.02 0.03 - 0.95 - - mov rdx, rcx + - - 0.36 0.02 - 0.62 0.50 0.50 cmp word ptr [rdi + rsi], -16192 + - - 0.99 0.99 - 0.02 - - cmove rax, rcx - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_dynamic_size.x86-64 index c5abb17f7ebab1..714b68c51734fe 100644 --- a/rust/zerocopy/benches/write_to_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64 @@ -3,8 +3,7 @@ bench_write_to_dynamic_size: push rbx push rax mov rbx, rcx - lea r14, [2*rsi + 5] - and r14, -2 + lea r14, [2*rsi + 4] cmp rcx, r14 jne .LBB5_2 mov rax, rdi diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca index 5b2c08a31a2976..2cc0a58c70fc7b 100644 --- a/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1900 +Instructions: 1800 Total Cycles: 2890 -Total uOps: 2500 +Total uOps: 2400 Dispatch Width: 4 -uOps Per Cycle: 0.87 -IPC: 0.66 -Block RThroughput: 6.3 +uOps Per Cycle: 0.83 +IPC: 0.62 +Block RThroughput: 6.0 Instruction Info: @@ -22,8 +22,7 @@ Instruction Info: 2 5 1.00 * push rbx 2 5 1.00 * push rax 1 1 0.33 mov rbx, rcx - 1 1 0.50 lea r14, [2*rsi + 5] - 1 1 0.33 and r14, -2 + 1 1 0.50 lea r14, [2*rsi + 4] 1 1 0.33 cmp rcx, r14 1 1 1.00 jne .LBB5_2 1 1 0.33 mov rax, rdi @@ -52,25 +51,24 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 4.66 4.64 4.00 4.70 4.00 3.00 + - - 4.47 4.01 4.00 4.52 4.00 3.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 1.00 - - 1.00 push r14 - - - - 1.00 - 1.00 - push rbx - - - - 1.00 - - 1.00 push rax - - - 0.02 0.97 - 0.01 - - mov rbx, rcx - - - 0.97 0.03 - - - - lea r14, [2*rsi + 5] - - - 0.63 0.35 - 0.02 - - and r14, -2 - - - 0.31 0.34 - 0.35 - - cmp rcx, r14 + - - - 0.99 - 0.01 - - mov rbx, rcx + - - 0.99 0.01 - - - - lea r14, [2*rsi + 4] + - - 0.02 0.95 - 0.03 - - cmp rcx, r14 - - - - - 1.00 - - jne .LBB5_2 - - - 0.33 0.33 - 0.34 - - mov rax, rdi - - - 0.36 0.31 - 0.33 - - mov rdi, rdx - - - 0.33 0.35 - 0.32 - - mov rsi, rax - - - 0.35 0.63 - 0.02 - - mov rdx, rbx + - - 0.48 0.04 - 0.48 - - mov rax, rdi + - - 0.51 0.01 - 0.48 - - mov rdi, rdx + - - 0.01 0.98 - 0.01 - - mov rsi, rax + - - 0.98 0.02 - - - - mov rdx, rbx - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL] - - - 0.65 0.35 - - - - cmp rbx, r14 - - - 0.69 - - 0.31 - - sete al + - - 0.94 0.03 - 0.03 - - cmp rbx, r14 + - - 0.52 - - 0.48 - - sete al - - 0.02 0.98 - - - - add rsp, 8 - - - - - - - 1.00 pop rbx - - - - - - 1.00 - pop r14 diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 index d7779c6c917873..63e5d5200962ef 100644 --- a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 @@ -3,8 +3,7 @@ bench_write_to_prefix_dynamic_size: push rbx push rax mov rbx, rcx - lea r14, [2*rsi + 5] - and r14, -2 + lea r14, [2*rsi + 4] cmp r14, rcx ja .LBB5_2 mov rax, rdi diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca index 4cebe24d4f3b90..e623ab3f302c42 100644 --- a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 1900 +Instructions: 1800 Total Cycles: 2890 -Total uOps: 2600 +Total uOps: 2500 Dispatch Width: 4 -uOps Per Cycle: 0.90 -IPC: 0.66 -Block RThroughput: 6.5 +uOps Per Cycle: 0.87 +IPC: 0.62 +Block RThroughput: 6.3 Instruction Info: @@ -22,8 +22,7 @@ Instruction Info: 2 5 1.00 * push rbx 2 5 1.00 * push rax 1 1 0.33 mov rbx, rcx - 1 1 0.50 lea r14, [2*rsi + 5] - 1 1 0.33 and r14, -2 + 1 1 0.50 lea r14, [2*rsi + 4] 1 1 0.33 cmp r14, rcx 1 1 1.00 ja .LBB5_2 1 1 0.33 mov rax, rdi @@ -52,26 +51,25 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 5.47 4.49 4.00 5.04 4.00 3.00 + - - 4.09 4.00 4.00 5.91 4.00 3.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 1.00 - - 1.00 push r14 - - - - 1.00 - 1.00 - push rbx - - - - 1.00 - - 1.00 push rax - - - 0.48 0.51 - 0.01 - - mov rbx, rcx - - - 0.51 0.49 - - - - lea r14, [2*rsi + 5] - - - 0.48 0.05 - 0.47 - - and r14, -2 - - - 0.48 0.49 - 0.03 - - cmp r14, rcx + - - 0.02 0.97 - 0.01 - - mov rbx, rcx + - - 0.04 0.96 - - - - lea r14, [2*rsi + 4] + - - 0.02 0.04 - 0.94 - - cmp r14, rcx - - - - - 1.00 - - ja .LBB5_2 - - - 0.04 0.47 - 0.49 - - mov rax, rdi - - - 0.49 0.03 - 0.48 - - mov rdi, rdx - - - 0.03 0.48 - 0.49 - - mov rsi, rax - - - 0.48 0.51 - 0.01 - - mov rdx, r14 + - - 0.96 0.01 - 0.03 - - mov rax, rdi + - - 0.03 0.93 - 0.04 - - mov rdi, rdx + - - 0.94 0.06 - - - - mov rsi, rax + - - 0.06 0.93 - 0.01 - - mov rdx, r14 - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL] - - - 0.51 0.49 - - - - cmp r14, rbx - - - 1.94 - - 0.06 - - setbe al - - - 0.03 0.97 - - - - add rsp, 8 + - - 0.93 0.05 - 0.02 - - cmp r14, rbx + - - 0.14 - - 1.86 - - setbe al + - - 0.95 0.05 - - - - add rsp, 8 - - - - - - - 1.00 pop rbx - - - - - - 1.00 - pop r14 - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 index 75f349562db65d..d13a9bb3b83cd9 100644 --- a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 @@ -3,8 +3,7 @@ bench_write_to_suffix_dynamic_size: push rbx push rax mov rbx, rcx - lea r14, [2*rsi + 5] - and r14, -2 + lea r14, [2*rsi + 4] sub rcx, r14 jb .LBB5_2 mov rax, rdi diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca index 95cb9dfe2eb78b..fb8fce317e0cd6 100644 --- a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 2000 +Instructions: 1900 Total Cycles: 2890 -Total uOps: 2600 +Total uOps: 2500 Dispatch Width: 4 -uOps Per Cycle: 0.90 -IPC: 0.69 -Block RThroughput: 6.5 +uOps Per Cycle: 0.87 +IPC: 0.66 +Block RThroughput: 6.3 Instruction Info: @@ -22,8 +22,7 @@ Instruction Info: 2 5 1.00 * push rbx 2 5 1.00 * push rax 1 1 0.33 mov rbx, rcx - 1 1 0.50 lea r14, [2*rsi + 5] - 1 1 0.33 and r14, -2 + 1 1 0.50 lea r14, [2*rsi + 4] 1 1 0.33 sub rcx, r14 1 1 1.00 jb .LBB5_2 1 1 0.33 mov rax, rdi @@ -53,27 +52,26 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 4.98 4.98 4.00 5.04 4.00 3.00 + - - 4.65 4.64 4.00 4.71 4.00 3.00 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - - 1.00 - - 1.00 push r14 - - - - 1.00 - 1.00 - push rbx - - - - 1.00 - - 1.00 push rax - - - 0.94 0.05 - 0.01 - - mov rbx, rcx - - - 0.06 0.94 - - - - lea r14, [2*rsi + 5] - - - 0.93 0.02 - 0.05 - - and r14, -2 - - - 0.05 0.94 - 0.01 - - sub rcx, r14 + - - 0.02 0.97 - 0.01 - - mov rbx, rcx + - - 0.97 0.03 - - - - lea r14, [2*rsi + 4] + - - 0.64 0.35 - 0.01 - - sub rcx, r14 - - - - - 1.00 - - jb .LBB5_2 - - - 0.02 0.04 - 0.94 - - mov rax, rdi - - - 0.03 0.97 - - - - add rdx, rcx - - - 0.95 0.05 - - - - mov rdi, rdx - - - 0.94 0.03 - 0.03 - - mov rsi, rax - - - 0.01 0.03 - 0.96 - - mov rdx, r14 + - - 0.35 0.31 - 0.34 - - mov rax, rdi + - - 0.32 0.68 - - - - add rdx, rcx + - - 0.32 0.34 - 0.34 - - mov rdi, rdx + - - 0.67 0.32 - 0.01 - - mov rsi, rax + - - 0.34 0.35 - 0.31 - - mov rdx, r14 - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL] - - - 0.05 0.94 - 0.01 - - cmp rbx, r14 - - - 0.97 - - 0.03 - - setae al - - - 0.03 0.97 - - - - add rsp, 8 + - - 0.35 0.31 - 0.34 - - cmp rbx, r14 + - - 0.65 - - 0.35 - - setae al + - - 0.02 0.98 - - - - add rsp, 8 - - - - - - - 1.00 pop rbx - - - - - - 1.00 - pop r14 - - - - - 1.00 - - ret diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64 b/rust/zerocopy/benches/zero_dynamic_size.x86-64 index 2b31ed644eaf8c..92e2748b1353c2 100644 --- a/rust/zerocopy/benches/zero_dynamic_size.x86-64 +++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64 @@ -1,5 +1,4 @@ bench_zero_dynamic_size: - lea rdx, [2*rsi + 5] - and rdx, -2 + lea rdx, [2*rsi + 4] xor esi, esi jmp qword ptr [rip + memset@GOTPCREL] diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca index 0b086a29b0ed64..7dee3b1de14197 100644 --- a/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca +++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca @@ -1,12 +1,12 @@ Iterations: 100 -Instructions: 400 -Total Cycles: 142 -Total uOps: 500 +Instructions: 300 +Total Cycles: 108 +Total uOps: 400 Dispatch Width: 4 -uOps Per Cycle: 3.52 -IPC: 2.82 -Block RThroughput: 1.3 +uOps Per Cycle: 3.70 +IPC: 2.78 +Block RThroughput: 1.0 Instruction Info: @@ -18,8 +18,7 @@ Instruction Info: [6]: HasSideEffects (U) [1] [2] [3] [4] [5] [6] Instructions: - 1 1 0.50 lea rdx, [2*rsi + 5] - 1 1 0.33 and rdx, -2 + 1 1 0.50 lea rdx, [2*rsi + 4] 1 0 0.25 xor esi, esi 2 6 1.00 * jmp qword ptr [rip + memset@GOTPCREL] @@ -37,11 +36,10 @@ Resources: Resource pressure per iteration: [0] [1] [2] [3] [4] [5] [6.0] [6.1] - - - 0.99 1.00 - 1.01 0.50 0.50 + - - 0.50 0.50 - 1.00 0.50 0.50 Resource pressure by instruction: [0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions: - - - 0.99 0.01 - - - - lea rdx, [2*rsi + 5] - - - - 0.99 - 0.01 - - and rdx, -2 + - - 0.50 0.50 - - - - lea rdx, [2*rsi + 4] - - - - - - - - xor esi, esi - - - - - 1.00 0.50 0.50 jmp qword ptr [rip + memset@GOTPCREL] diff --git a/rust/zerocopy/src/byte_slice.rs b/rust/zerocopy/src/byte_slice.rs index b7f85098dbc4c0..26fcc088aecf36 100644 --- a/rust/zerocopy/src/byte_slice.rs +++ b/rust/zerocopy/src/byte_slice.rs @@ -427,8 +427,12 @@ mod tests { let borrow_mut = cell.borrow_mut(); let slice_ref_mut: cell::RefMut<'_, [u8]> = cell::RefMut::map(borrow_mut, |a| &mut a[..]); // SAFETY: 2 is within bounds of [1, 2, 3, 4] - let (l, r) = unsafe { slice_ref_mut.split_at_unchecked(2) }; + let (mut l, mut r) = unsafe { slice_ref_mut.split_at_unchecked(2) }; assert_eq!(*l, [1, 2]); assert_eq!(*r, [3, 4]); + l[0] = 5; + r[0] = 6; + drop((l, r)); + assert_eq!(*cell.borrow(), [5, 2, 6, 4]); } } diff --git a/rust/zerocopy/src/byteorder.rs b/rust/zerocopy/src/byteorder.rs index c761d5728320bf..6189a6e7f77c2b 100644 --- a/rust/zerocopy/src/byteorder.rs +++ b/rust/zerocopy/src/byteorder.rs @@ -87,7 +87,7 @@ use super::*; pub trait ByteOrder: Copy + Clone + Debug + Display + Eq + PartialEq + Ord + PartialOrd + Hash + private::Sealed { - #[doc(hidden)] + /// A value-level representation of byte order. const ORDER: Order; } @@ -98,11 +98,12 @@ mod private { impl Sealed for super::LittleEndian {} } -#[allow(missing_copy_implementations, missing_debug_implementations)] -#[doc(hidden)] -#[derive(PartialEq)] +/// A value-level representation of [`ByteOrder`]. +#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)] pub enum Order { + /// A value-level representation of [`BigEndian`]. BigEndian, + /// A value-level representation of [`LittleEndian`]. LittleEndian, } @@ -1235,7 +1236,7 @@ mod tests { impl_traits!(I32, i32, signed); impl_traits!(I64, i64, signed); impl_traits!(I128, i128, signed); - impl_traits!(Isize, isize, unsigned); + impl_traits!(Isize, isize, signed); impl_traits!(F32, f32, signed, @float); impl_traits!(F64, f64, signed, @float); @@ -1383,6 +1384,18 @@ mod tests { call_for_all_types!(test_non_native, NonNativeEndian); } + #[test] + fn test_float_bit_patterns() { + for bits in [0u32, 0x8000_0000, 0x7F80_0000, 0xFF80_0000, 0x7FC0_1234] { + let be = F32::::new(f32::from_bits(bits)); + let le = F32::::new(f32::from_bits(bits)); + assert_eq!(be.to_bytes(), bits.to_be_bytes()); + assert_eq!(le.to_bytes(), bits.to_le_bytes()); + assert_eq!(be.get().to_bits(), bits); + assert_eq!(le.get().to_bits(), bits); + } + } + #[test] fn test_ops_impls() { // Test implementations of traits in `core::ops`. Some of these are @@ -1551,6 +1564,17 @@ mod tests { test!(@unary Not, not, call_for_signed_types, call_for_unsigned_types); test!(@unary Neg, neg, call_for_signed_types, call_for_float_types); + + for shift in [0u64, 1, 31, 63] { + let n = 0x0123_4567_89AB_CDEFu64; + let shift_u32: u32 = shift.try_into().unwrap(); + let shl = n.checked_shl(shift_u32).unwrap(); + let shr = n.checked_shr(shift_u32).unwrap(); + assert_eq!(core::ops::Shl::shl(U64::::new(n), shift).get(), shl); + assert_eq!(core::ops::Shr::shr(U64::::new(n), shift).get(), shr); + assert_eq!(core::ops::Shl::shl(U64::::new(n), shift).get(), shl); + assert_eq!(core::ops::Shr::shr(U64::::new(n), shift).get(), shr); + } } #[test] @@ -1579,6 +1603,13 @@ mod tests { assert!(val_be <= val_be); assert_eq!(val_be.cmp(&val_be), core::cmp::Ordering::Equal); + let low = U16::::new(255); + let high = U16::::new(256); + assert!(low < high); + assert!(low < 256u16); + assert_eq!(low.cmp(&high), core::cmp::Ordering::Less); + assert!(I16::::new(-1) < I16::::new(0)); + // PartialOrd with native assert!(val_be == 1u16); assert!(val_be >= 1u16); diff --git a/rust/zerocopy/src/error.rs b/rust/zerocopy/src/error.rs index 5eb30de934f38e..98399bab7e970f 100644 --- a/rust/zerocopy/src/error.rs +++ b/rust/zerocopy/src/error.rs @@ -199,10 +199,8 @@ impl From, S, V>) -> ConvertError { match err { - ConvertError::Alignment(e) => { - #[allow(unreachable_code)] - return ConvertError::Alignment(Infallible::from(e)); - } + #[allow(unreachable_code)] + ConvertError::Alignment(e) => match Infallible::from(e) {}, ConvertError::Size(e) => ConvertError::Size(e), ConvertError::Validity(e) => ConvertError::Validity(e), } @@ -237,7 +235,7 @@ impl fmt::Display for Convert } #[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))] -#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))] +#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))] impl Error for ConvertError where A: fmt::Display + fmt::Debug, @@ -410,7 +408,7 @@ where } #[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))] -#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))] +#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))] impl Error for AlignmentError where Src: Deref, @@ -571,7 +569,7 @@ where } #[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))] -#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))] +#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))] impl Error for SizeError where Src: Deref, @@ -707,7 +705,7 @@ where } #[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))] -#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))] +#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))] impl Error for ValidityError where Dst: KnownLayout + TryFromBytes {} impl From> diff --git a/rust/zerocopy/src/impls.rs b/rust/zerocopy/src/impls.rs index 62e234c5202b58..01b509b9826444 100644 --- a/rust/zerocopy/src/impls.rs +++ b/rust/zerocopy/src/impls.rs @@ -111,7 +111,7 @@ assert_unaligned!(bool); // pattern 0x01. const _: () = unsafe { unsafe_impl!(=> TryFromBytes for bool; |byte| { - let byte = byte.transmute_with::(); + let byte = byte.transmute_with::(); *byte.unaligned_as_ref() < 2 }) }; @@ -140,7 +140,7 @@ const _: () = unsafe { unsafe_impl!(char: Immutable, FromZeros, IntoBytes) }; // `char`. const _: () = unsafe { unsafe_impl!(=> TryFromBytes for char; |c| { - let c = c.transmute_with::, invariant::Valid, CastSizedExact, BecauseImmutable>(); + let c = c.transmute_with::, invariant::Safe, CastSizedExact, BecauseImmutable>(); let c = c.read().into_inner(); char::from_u32(c).is_some() }); @@ -173,7 +173,7 @@ const _: () = unsafe { unsafe_impl!(str: Immutable, FromZeros, IntoBytes, Unalig // Returns `Err` if the slice is not UTF-8. const _: () = unsafe { unsafe_impl!(=> TryFromBytes for str; |c| { - let c = c.transmute_with::<[u8], invariant::Valid, CastUnsized, BecauseImmutable>(); + let c = c.transmute_with::<[u8], invariant::Safe, CastUnsized, BecauseImmutable>(); let c = c.unaligned_as_ref(); core::str::from_utf8(c).is_ok() }) @@ -183,7 +183,7 @@ macro_rules! unsafe_impl_try_from_bytes_for_nonzero { ($($nonzero:ident[$prim:ty]),*) => { $( unsafe_impl!(=> TryFromBytes for $nonzero; |n| { - let n = n.transmute_with::, invariant::Valid, CastSizedExact, BecauseImmutable>(); + let n = n.transmute_with::, invariant::Safe, CastSizedExact, BecauseImmutable>(); $nonzero::new(n.read().into_inner()).is_some() }); )* @@ -421,15 +421,15 @@ mod atomics { ($($($tyvar:ident)? => $atomic:ty [$prim:ty]),*) => {{ crate::util::macros::__unsafe(); - use crate::pointer::{SizeEq, TransmuteFrom, invariant::Valid}; + use crate::pointer::{SizeEq, TransmuteFrom, invariant::Safe}; $( // SAFETY: The caller promised that `$atomic` and `$prim` have // the same size and bit validity. - unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for $prim {} + unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Safe, Safe> for $prim {} // SAFETY: The caller promised that `$atomic` and `$prim` have // the same size and bit validity. - unsafe impl<$($tyvar)?> TransmuteFrom<$prim, Valid, Valid> for $atomic {} + unsafe impl<$($tyvar)?> TransmuteFrom<$prim, Safe, Safe> for $atomic {} impl<$($tyvar)?> SizeEq> for ReadOnly<$prim> { type CastFrom = $crate::pointer::cast::CastSizedExact; @@ -444,9 +444,9 @@ mod atomics { // `UnsafeCell` has the same in-memory representation as // its inner type `T`. A consequence of this guarantee is that // it is possible to convert between `T` and `UnsafeCell`. - unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for core::cell::UnsafeCell<$prim> {} + unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Safe, Safe> for core::cell::UnsafeCell<$prim> {} // SAFETY: See previous safety comment. - unsafe impl<$($tyvar)?> TransmuteFrom, Valid, Valid> for $atomic {} + unsafe impl<$($tyvar)?> TransmuteFrom, Safe, Safe> for $atomic {} )* }}; } @@ -758,7 +758,7 @@ const _: () = { pub enum value {} // SAFETY: See safety comment on `ProjectToTag`. - unsafe impl HasTag for ManuallyDrop { + unsafe impl HasTag for ManuallyDrop { #[inline] fn only_derive_is_allowed_to_implement_this_trait() where @@ -796,8 +796,8 @@ const _: () = { // private field, and because it is the name it is referred to in the public // documentation of `ManuallyDrop::new`, `ManuallyDrop::into_inner`, // `ManuallyDrop::take` and `ManuallyDrop::drop`. - unsafe impl - HasField + unsafe impl + HasField for ManuallyDrop { #[inline] @@ -849,7 +849,7 @@ impl_for_transmute_from!(T: ?Sized + IntoBytes => IntoBytes for UnsafeCell[T] const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for UnsafeCell) }; assert_unaligned!(UnsafeCell<()>, UnsafeCell); -// SAFETY: See safety comment in `is_bit_valid` impl. +// SAFETY: See safety comment in `is_safe` impl. unsafe impl TryFromBytes for UnsafeCell { #[allow(clippy::missing_inline_in_public_items)] fn only_derive_is_allowed_to_implement_this_trait() @@ -859,11 +859,11 @@ unsafe impl TryFromBytes for UnsafeCell { } #[inline(always)] - fn is_bit_valid(candidate: Maybe<'_, Self, A>) -> bool + fn is_safe(candidate: Maybe<'_, Self, A>) -> bool where A: invariant::Alignment, { - T::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>()) + T::is_safe(candidate.transmute::<_, _, BecauseImmutable>()) } } @@ -897,10 +897,10 @@ const _: () = unsafe { let c: Ptr<'_, ReadOnly<[T]>, _> = c.cast::<_, crate::pointer::cast::CastUnsized, _>(); // Note that this call may panic, but it would still be sound even if it - // did. `is_bit_valid` does not promise that it will not panic (in fact, - // it explicitly warns that it's a possibility), and we have not - // violated any safety invariants that we must fix before returning. - <[T] as TryFromBytes>::is_bit_valid(c) + // did. `is_safe` does not promise that it will not panic (in fact, it + // explicitly warns that it's a possibility), and we have not violated + // any safety invariants that we must fix before returning. + <[T] as TryFromBytes>::is_safe(c) }); unsafe_impl!(const N: usize, T: FromZeros => FromZeros for [T; N]); unsafe_impl!(const N: usize, T: FromBytes => FromBytes for [T; N]); @@ -924,14 +924,14 @@ const _: () = unsafe { // // In other words, the layout of a `[T] is a sequence of `T`s laid out // back-to-back with no bytes in between. If all elements in `candidate` - // are `is_bit_valid`, so too is `candidate`. + // are `is_safe`, so too is `candidate`. // // Note that any of the below calls may panic, but it would still be - // sound even if it did. `is_bit_valid` does not promise that it will - // not panic (in fact, it explicitly warns that it's a possibility), and - // we have not violated any safety invariants that we must fix before + // sound even if it did. `is_safe` does not promise that it will not + // panic (in fact, it explicitly warns that it's a possibility), and we + // have not violated any safety invariants that we must fix before // returning. - c.iter().all(::is_bit_valid) + c.iter().all(::is_safe) }); unsafe_impl!(T: FromZeros => FromZeros for [T]); unsafe_impl!(T: FromBytes => FromBytes for [T]); @@ -946,7 +946,7 @@ const _: () = unsafe { // null pointers, so this is not a footgun. // - `TryFromBytes`: By the same reasoning as for `FromZeroes`, we can implement // `TryFromBytes` for thin pointers provided that -// [`TryFromByte::is_bit_valid`] only produces `true` for zeroed bytes. +// [`TryFromBytes::is_safe`] only produces `true` for zeroed bytes. // // NOTE(#170): Implementing `FromBytes` and `IntoBytes` for raw pointers would // be sound, but carries provenance footguns. We want to support `FromBytes` and @@ -1012,11 +1012,11 @@ mod tuples { // SAFETY: If all fields of the tuple `Self` are `Immutable`, so too is `Self`. unsafe_impl!($($head_T: Immutable,)* $next_T: Immutable => Immutable for ($($head_T,)* $next_T,)); - // SAFETY: If all fields in `c` are `is_bit_valid`, so too is `c`. + // SAFETY: If all fields in `c` are `is_safe`, so too is `c`. unsafe_impl!($($head_T: TryFromBytes,)* $next_T: TryFromBytes => TryFromBytes for ($($head_T,)* $next_T,); |c| { let mut c = c; - $(TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($head_I) }>())) &&)* - TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($next_I) }>())) + $(TryFromBytes::is_safe(into_inner!(c.reborrow().project::())) &&)* + TryFromBytes::is_safe(into_inner!(c.reborrow().project::())) }); // SAFETY: If all fields in `Self` are `FromZeros`, so too is `Self`. @@ -1026,7 +1026,9 @@ mod tuples { unsafe_impl!($($head_T: FromBytes,)* $next_T: FromBytes => FromBytes for ($($head_T,)* $next_T,)); // SAFETY: See safety comment on `ProjectToTag`. - unsafe impl<$($head_T,)* $next_T> crate::HasTag for ($($head_T,)* $next_T,) { + unsafe impl crate::HasTag + for ($($head_T,)* $next_T,) + { #[inline] fn only_derive_is_allowed_to_implement_this_trait() where @@ -1077,7 +1079,8 @@ mod tuples { // - `()` has the same visibility as the `.$CurrI` field (ie, `.0`, // `.1`, etc) // - `Type` has the same type as `$CurrI`; i.e., `$CurrT`. - unsafe impl<$($AllT),+> crate::HasField< + unsafe impl crate::HasField< + Client, (), { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($CurrI)} @@ -1103,7 +1106,8 @@ mod tuples { } // SAFETY: See comments on items. - unsafe impl crate::ProjectField< + unsafe impl crate::ProjectField< + Client, (), (Aliasing, Alignment, crate::invariant::Uninit), { crate::STRUCT_VARIANT_ID }, @@ -1129,7 +1133,8 @@ mod tuples { } // SAFETY: See comments on items. - unsafe impl crate::ProjectField< + unsafe impl crate::ProjectField< + Client, (), (Aliasing, Alignment, crate::invariant::Initialized), { crate::STRUCT_VARIANT_ID }, @@ -1155,9 +1160,10 @@ mod tuples { } // SAFETY: See comments on items. - unsafe impl crate::ProjectField< + unsafe impl crate::ProjectField< + Client, (), - (Aliasing, Alignment, crate::invariant::Valid), + (Aliasing, Alignment, crate::invariant::Safe), { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($CurrI)} > for ($($AllT,)+) @@ -1174,7 +1180,7 @@ mod tuples { // SAFETY: Tuples are product types whose fields are // well-aligned, so projection preserves both the alignment and // validity invariants of the outer pointer. - type Invariants = (Aliasing, Alignment, crate::invariant::Valid); + type Invariants = (Aliasing, Alignment, crate::invariant::Safe); // SAFETY: Tuples are product types and so projection is infallible; type Error = core::convert::Infallible; @@ -1389,10 +1395,9 @@ mod tests { #[test] fn test_impls() { - // A type that can supply test cases for testing - // `TryFromBytes::is_bit_valid`. All types passed to `assert_impls!` - // must implement this trait; that macro uses it to generate runtime - // tests for `TryFromBytes` impls. + // A type that can supply test cases for testing `TryFromBytes::is_safe`. + // All types passed to `assert_impls!` must implement this trait; that + // macro uses it to generate runtime tests for `TryFromBytes` impls. // // All `T: FromBytes` types are provided with a blanket impl. Other // types must implement `TryFromBytesTestable` directly (ie using @@ -1574,19 +1579,15 @@ mod tests { pub(super) struct AutorefWrapper(pub(super) PhantomData); - pub(super) trait TestIsBitValidShared { + pub(super) trait TestIsSafeShared { #[allow(clippy::needless_lifetimes)] - fn test_is_bit_valid_shared<'ptr>(&self, candidate: Maybe<'ptr, T>) - -> Option; + fn test_is_safe_shared<'ptr>(&self, candidate: Maybe<'ptr, T>) -> Option; } - impl TestIsBitValidShared for AutorefWrapper { + impl TestIsSafeShared for AutorefWrapper { #[allow(clippy::needless_lifetimes)] - fn test_is_bit_valid_shared<'ptr>( - &self, - candidate: Maybe<'ptr, T>, - ) -> Option { - Some(T::is_bit_valid(candidate)) + fn test_is_safe_shared<'ptr>(&self, candidate: Maybe<'ptr, T>) -> Option { + Some(T::is_safe(candidate)) } } @@ -1690,12 +1691,12 @@ mod tests { #[allow(unused, non_local_definitions)] impl AutorefWrapper<$ty> { #[allow(clippy::needless_lifetimes)] - fn test_is_bit_valid_shared<'ptr>( + fn test_is_safe_shared<'ptr>( &mut self, candidate: Maybe<'ptr, $ty>, ) -> Option { assert_on_allowlist!( - test_is_bit_valid_shared($ty): + test_is_safe_shared($ty): ManuallyDrop>, ManuallyDrop<[UnsafeCell]>, ManuallyDrop<[UnsafeCell]>, @@ -1799,9 +1800,9 @@ mod tests { // necessarily `IntoBytes`, but that's the corner we've // backed ourselves into by using `Ptr::from_ref`. let c = unsafe { c.assume_initialized() }; - let res = w.test_is_bit_valid_shared(c); + let res = w.test_is_safe_shared(c); if let Some(res) = res { - assert!(res, "{}::is_bit_valid (shared `Ptr`): got false, expected true", stringify!($ty)); + assert!(res, "{}::is_safe (shared `Ptr`): got false, expected true", stringify!($ty)); } let c = Ptr::from_mut(&mut *val); @@ -1810,8 +1811,8 @@ mod tests { // necessarily `IntoBytes`, but that's the corner we've // backed ourselves into by using `Ptr::from_ref`. let mut c = unsafe { c.assume_initialized() }; - let res = <$ty as TryFromBytes>::is_bit_valid(c.reborrow_shared()); - assert!(res, "{}::is_bit_valid (exclusive `Ptr`): got false, expected true", stringify!($ty)); + let res = <$ty as TryFromBytes>::is_safe(c.reborrow_shared()); + assert!(res, "{}::is_safe (exclusive `Ptr`): got false, expected true", stringify!($ty)); // `bytes` is `Some(val.as_bytes())` if `$ty: IntoBytes + // Immutable` and `None` otherwise. @@ -2237,11 +2238,11 @@ mod tests { assert_impls!(ManuallyDrop: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned); // This test is important because it allows us to test our hand-rolled - // implementation of ` as TryFromBytes>::is_bit_valid`. + // implementation of ` as TryFromBytes>::is_safe`. assert_impls!(ManuallyDrop: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes); assert_impls!(ManuallyDrop<[u8]>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned); // This test is important because it allows us to test our hand-rolled - // implementation of ` as TryFromBytes>::is_bit_valid`. + // implementation of ` as TryFromBytes>::is_safe`. assert_impls!(ManuallyDrop<[bool]>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes); assert_impls!(ManuallyDrop: !Immutable, !TryFromBytes, !KnownLayout, !FromZeros, !FromBytes, !IntoBytes, !Unaligned); assert_impls!(ManuallyDrop<[NotZerocopy]>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned); @@ -2255,14 +2256,14 @@ mod tests { assert_impls!(Wrapping: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned); // This test is important because it allows us to test our hand-rolled - // implementation of ` as TryFromBytes>::is_bit_valid`. + // implementation of ` as TryFromBytes>::is_safe`. assert_impls!(Wrapping: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes); assert_impls!(Wrapping: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned); assert_impls!(Wrapping>: KnownLayout, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned, !Immutable); assert_impls!(Unalign: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned); // This test is important because it allows us to test our hand-rolled - // implementation of ` as TryFromBytes>::is_bit_valid`. + // implementation of ` as TryFromBytes>::is_safe`. assert_impls!(Unalign: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes); assert_impls!(Unalign: KnownLayout, Unaligned, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes); diff --git a/rust/zerocopy/src/layout.rs b/rust/zerocopy/src/layout.rs index b1fa0cd436db8d..5d5598cd0cfbba 100644 --- a/rust/zerocopy/src/layout.rs +++ b/rust/zerocopy/src/layout.rs @@ -30,6 +30,7 @@ const POINTER_WIDTH_BITS: usize = mem::size_of::() * 8; #[allow(missing_debug_implementations, missing_copy_implementations)] #[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))] #[derive(Copy, Clone)] +#[cfg_attr(kani, derive(kani::Arbitrary))] pub struct DstLayout { pub(crate) align: NonZeroUsize, pub(crate) size_info: SizeInfo, @@ -42,29 +43,680 @@ pub struct DstLayout { #[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))] #[derive(Copy, Clone)] +#[cfg_attr(kani, derive(kani::Arbitrary))] pub(crate) enum SizeInfo { Sized { size: usize }, SliceDst(TrailingSliceLayout), } +/// The alignment and phase of a rounded size calculation. +/// +/// If `encoded` is the stored non-zero integer, then its highest set bit is a +/// power-of-two alignment `align`, and its remaining lower bits are a `phase` +/// in `0..align`. Thus every non-zero bit pattern is a valid encoding, and: +/// +/// ```text +/// encoded = align + phase +/// ``` +#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))] +#[repr(transparent)] +#[derive(Copy, Clone)] +#[cfg_attr(kani, derive(kani::Arbitrary))] +pub(crate) struct RoundingAlignAndPhase(NonZeroUsize); + +impl RoundingAlignAndPhase { + /// Encodes `align` and `phase`. + /// + /// # Panics + /// + /// Panics if `align` is not a power of two and `phase` is not less than + /// `align`. + #[inline(always)] + #[cfg_attr(kani, kani::requires(align.is_power_of_two() && phase < align.get()))] + #[cfg_attr(kani, kani::ensures(|result| result.0.get() == align.get() + phase))] + pub(crate) const fn new(align: NonZeroUsize, phase: usize) -> Self { + #[cfg(kani)] + #[kani::proof_for_contract(RoundingAlignAndPhase::new)] + #[kani::solver(kissat)] + fn proof() { + RoundingAlignAndPhase::new(kani::any(), kani::any()); + } + + const_assert!(align.get().is_power_of_two()); + const_assert!(phase < align.get()); + + // Since `align` is a power of two and `phase < align`, their set bits + // are disjoint. The result is therefore non-zero and losslessly stores + // both components. + let encoded = align.get() | phase; + match NonZeroUsize::new(encoded) { + Some(encoded) => Self(encoded), + None => const_unreachable!(), + } + } + + /// Decodes the alignment and phase. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&(align, phase)| { + align.is_power_of_two() && phase < align.get() + && align.get().checked_add(phase) == Some(self.0.get()) + }))] + pub(crate) const fn components(self) -> (NonZeroUsize, usize) { + #[cfg(kani)] + #[kani::proof_for_contract(RoundingAlignAndPhase::components)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().components(); + } + // `leading_zeros <= POINTER_WIDTH_BITS - 1` because the encoded value + // is non-zero. Converting `leading_zeros` to `usize` cannot truncate + // because it is at most the number of bits in a `usize`. + #[allow( + clippy::arithmetic_side_effects, + clippy::as_conversions, + clippy::needless_nonzero_get + )] + let shift = (POINTER_WIDTH_BITS - 1) - (self.0.get().leading_zeros() as usize); + #[allow(clippy::arithmetic_side_effects)] + let align = 1usize << shift; + let align = match NonZeroUsize::new(align) { + Some(align) => align, + None => const_unreachable!(), + }; + // `align` is the highest set bit, so XOR removes exactly that bit and + // leaves a value in `0..align`. + let phase = self.0.get() ^ align.get(); + (align, phase) + } + + /// Decodes the alignment, which is the highest set bit of `self`. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|align| { + align.is_power_of_two() && align.get() <= self.0.get() + && self.0.get() - align.get() < align.get() + }))] + pub(crate) const fn align(self) -> NonZeroUsize { + #[cfg(kani)] + #[kani::proof_for_contract(RoundingAlignAndPhase::align)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().align(); + } + + self.components().0 + } +} + #[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))] #[derive(Copy, Clone)] +#[cfg_attr(kani, derive(kani::Arbitrary))] pub(crate) struct TrailingSliceLayout { - // The offset of the first byte of the trailing slice field. Note that this - // is NOT the same as the minimum size of the type. For example, consider - // the following type: - // - // struct Foo { - // a: u16, - // b: u8, - // c: [u8], - // } - // - // In `Foo`, `c` is at byte offset 3. When `c.len() == 0`, `c` is followed - // by a padding byte. + /// The offset of the first byte of the trailing slice field. Note that this + /// is NOT the same as the minimum size of the type. For example, consider + /// the following type: + /// + /// ```ignore + /// #[repr(C, align(2))] + /// struct Foo { + /// header: u16, + /// tag: u8, + /// tail: [u8], + /// } + /// ``` + /// + /// In `Foo`, `tail` is at byte offset 3. When `tail.len() == 0`, `tail` is + /// followed by a padding byte. pub(crate) offset: usize, - // The size of the element type of the trailing slice field. + /// The size of the element type of the trailing slice field. pub(crate) elem_size: E, + /// The trailing slice's byte offset plus fixed trailing padding, minus the + /// rounding phase. + /// + /// Fixed trailing padding excludes padding added by the final alignment + /// rounding. The rounding alignment and phase are stored in + /// `size_rounding_align_and_phase`. This value is independent of the slice + /// length and can be smaller than, equal to, or larger than `offset`. + /// + /// For a completed layout, this value is a multiple of `DstLayout::align`. + /// The minimum object size equals this value when the phase is zero; + /// otherwise, it exceeds this value by one rounding-alignment unit. + pub(crate) size_base: usize, + /// The alignment and byte phase of the rounding applied to the trailing + /// slice's contribution to object size. + /// + /// The alignment `size_align` is a nonzero power of two. The phase + /// `size_phase` is the fixed byte contribution inside that rounding, in + /// `0..size_align`. Both are independent of the trailing slice's length. + /// For `elems` trailing elements, the enclosing object's size is: + /// + /// ```text + /// size_base + round_up(size_phase + elems * elem_size, size_align) + /// ``` + /// + /// For a completed layout, `size_align` is at least `DstLayout::align`. + /// Packing can reduce the enclosing type's alignment and the alignment + /// used to place its trailing field without removing padding required by + /// a nested DST's own layout. For example, a `repr(packed(2))` wrapper + /// around a DST with alignment 4 can have an object alignment of 2 and a + /// rounding alignment of 4. + /// + /// The rounding parameters account for padding preserved within nested + /// fields. Their rounding origin can differ from the physical start of + /// the trailing slice, so the phase need not equal `offset % size_align`. + pub(crate) size_rounding_align_and_phase: RoundingAlignAndPhase, +} + +impl TrailingSliceLayout { + /// Produces `self.size_base` rounded down to a multiple of `size_align`, + /// plus `size_phase`, where both components come from + /// `self.size_rounding_align_and_phase`. + /// + /// This value need not equal `self.offset`, the trailing slice's byte + /// offset. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&offset| { + let (align, phase) = self.size_rounding_align_and_phase.components(); + offset == util::round_down_to_next_multiple_of_alignment(self.size_base, align) + phase + }))] + const fn size_offset(&self) -> usize { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::::size_offset)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().size_offset(); + } + + let (size_align, size_phase) = self.size_rounding_align_and_phase.components(); + let aligned_base = + util::round_down_to_next_multiple_of_alignment(self.size_base, size_align); + // The aligned base and phase have disjoint set bits. + aligned_base | size_phase + } + + /// Produces the largest trailing-slice byte capacity whose complete, padded + /// object fits in `available_bytes`, or `None` if the object does not fit + /// even with an empty trailing slice. + /// + /// This capacity is independent of `elem_size`: it need not represent a + /// whole number of elements, even for a valid Rust layout. For example, a + /// slice of `[u8; 3]` elements has three-byte elements and alignment 1. + /// With `available_bytes = 4`, this method produces `Some(4)`, although + /// only one whole element, occupying three bytes, fits. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&result| { + let (align, phase) = self.size_rounding_align_and_phase.components(); + result == available_bytes.checked_sub(self.size_base) + .map(|bytes| util::round_down_to_next_multiple_of_alignment(bytes, align)) + .and_then(|rounded| rounded.checked_sub(phase)) + }))] + const fn max_trailing_bytes(&self, available_bytes: usize) -> Option { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::::max_trailing_bytes)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().max_trailing_bytes(kani::any()); + } + + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::::max_trailing_bytes)] + #[kani::solver(kissat)] + fn proof_nonzero() { + kani::any::>().max_trailing_bytes(kani::any()); + } + + let (size_align, size_phase) = self.size_rounding_align_and_phase.components(); + + // Interpret the normalized size formula over the nonnegative integers: + // + // object_size(trailing_bytes) + // = self.size_base + // + round_up(size_phase + trailing_bytes, size_align). + // + // We seek the largest byte count with + // `object_size(trailing_bytes) <= available_bytes`. + // + // At zero trailing bytes, the input to rounding is just `size_phase`. + // Since `0 <= size_phase < size_align`, the least alignment multiple at + // or above it is zero for a zero phase, and `size_align` otherwise. + // Thus `rounded_phase = round_up(size_phase, size_align)` is exactly + // the rounded contribution of an empty tail. + let rounded_phase = if size_phase == 0 { 0 } else { size_align.get() }; + + // Adding the fixed base gives `minimum_size = object_size(0)`. The size + // formula is nondecreasing in the trailing byte count, so this is the + // minimum object size. If this sum exceeds `usize::MAX`, it also + // exceeds `available_bytes`, and no tail can fit. + let minimum_size = match self.size_base.checked_add(rounded_phase) { + Some(size) => size, + None => return None, + }; + + // Subtracting the minimum size gives the budget for increasing the + // rounded contribution. Failure means even the minimum size exceeds the + // available space. Success establishes `minimum_size + extra_bytes = + // available_bytes`. + let extra_bytes = match available_bytes.checked_sub(minimum_size) { + Some(bytes) => bytes, + None => return None, + }; + + // The rounded contribution can increase only by multiples of + // `size_align`. Its largest permitted increase is therefore + // `aligned_bytes = round_down(extra_bytes, size_align)`, with: + // + // aligned_bytes <= extra_bytes < aligned_bytes + size_align. + // + // Hence `rounded_phase + aligned_bytes` is the largest alignment + // multiple that fits after the base: adding the base gives at most + // `minimum_size + extra_bytes = available_bytes`, while the next + // alignment multiple would exceed that budget. + let aligned_bytes = util::round_down_to_next_multiple_of_alignment(extra_bytes, size_align); + + // Rounding an empty tail adds `rounded_phase - size_phase` bytes. This + // is `initial_padding`: the tail can use these bytes without increasing + // the rounded contribution. The two cases for `rounded_phase` give `0 + // <= initial_padding < size_align`, so the subtraction is + // representable. + #[allow(clippy::arithmetic_side_effects)] + let initial_padding = rounded_phase - size_phase; + + // For any nonnegative candidate byte count, the aligned limit above + // gives the following equivalences: + // + // object_size(candidate_bytes) <= available_bytes + // iff round_up(size_phase + candidate_bytes, size_align) + // <= rounded_phase + aligned_bytes + // iff size_phase + candidate_bytes + // <= rounded_phase + aligned_bytes + // iff candidate_bytes <= aligned_bytes + initial_padding. + // + // Removing `round_up` is valid because its upper bound is an alignment + // multiple. Thus `aligned_bytes + initial_padding` is exactly the + // largest tail that fits. Since `aligned_bytes` is a multiple of the + // power-of-two `size_align` and `initial_padding < size_align`, their + // set bits are disjoint. Bitwise OR therefore equals their sum, with no + // carry or overflow. + let trailing_bytes = aligned_bytes | initial_padding; + Some(trailing_bytes) + } +} + +impl TrailingSliceLayout { + /// Returns the difference between the complete object size and the + /// trailing-slice end for `elems` elements, modulo `usize::MAX + 1`. + /// + /// If the object size fits in `usize` and contains the trailing slice, + /// this is the exact trailing padding. Other inputs need not describe a + /// Rust type; their result is still the modular difference. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&padding| { + let (align, phase) = self.size_rounding_align_and_phase.components(); + let trailing_bytes = elems.wrapping_mul(self.elem_size); + let mask = align.get() - 1; + let rounded = phase.wrapping_add(trailing_bytes).wrapping_add(mask) & !mask; + let object_size = self.size_base.wrapping_add(rounded); + padding == object_size.wrapping_sub(self.offset.wrapping_add(trailing_bytes)) + }))] + pub(crate) const fn padding_for_elems(self, elems: usize) -> usize { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::padding_for_elems)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().padding_for_elems(kani::any()); + } + + let (size_align, size_phase) = self.size_rounding_align_and_phase.components(); + + // Since `size_align` is a nonzero power of two, subtracting one gives + // a mask selecting precisely the remainder modulo `size_align`. + #[allow(clippy::arithmetic_side_effects)] + let size_mask = size_align.get() - 1; + + // Removing whole alignment multiples from each element's size + // preserves the trailing-byte count modulo `size_align`. If elements + // occupy whole alignment units, this is zero, making the padding's + // independence from the element count explicit to the optimizer. + let elem_remainder = self.elem_size & size_mask; + + // The product is congruent to `elems * self.elem_size` modulo + // `size_align`. Wrapping preserves that congruence because every + // power-of-two `size_align` divides `usize::MAX + 1`. Masking then + // gives the exact trailing-byte remainder in `0..size_align`. + let trailing_remainder = elems.wrapping_mul(elem_remainder) & size_mask; + + // Adding `size_phase` gives the same remainder as the mathematical + // `size_phase + elems * self.elem_size`. Reducing before adding the + // phase also exposes this periodicity to the optimizer. + let rounding_input = size_phase.wrapping_add(trailing_remainder); + + // `rounding_padding` is the unique value in `0..size_align` that + // rounds `rounding_input` up to an alignment multiple. Since only + // the remainder determines this value, it is also the padding added + // by `round_up(size_phase + elems * self.elem_size, size_align)`. + let rounding_padding = util::padding_needed_for(rounding_input, size_align); + + // Write `trailing_bytes = elems * self.elem_size` over mathematical + // integers. Subtracting the trailing-slice end from the complete + // object size cancels `trailing_bytes`: + // + // size_base + round_up(size_phase + trailing_bytes, size_align) + // - (offset + trailing_bytes) + // = size_base + size_phase - offset + rounding_padding. + // + // Wrapping computes this difference modulo `usize::MAX + 1`, even + // if an intermediate value or the difference is unrepresentable. + // If the object size fits in `usize` and contains the trailing slice, + // the difference is in `0..=usize::MAX`; its residue is then the + // exact number of padding bytes after the trailing slice. + self.size_base + .wrapping_add(size_phase) + .wrapping_sub(self.offset) + .wrapping_add(rounding_padding) + } + + /// Computes the size described by this layout for `elems` trailing slice + /// elements. + /// + /// Returns `None` if the element-byte count or complete rounded size + /// cannot be represented as a `usize`. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&size| { + size == elems.checked_mul(self.elem_size) + .and_then(|bytes| proofs::slice_dst_size_for_trailing_bytes(self, bytes)) + }))] + pub(crate) const fn size_for_elems(self, elems: usize) -> Option { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::size_for_elems)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().size_for_elems(kani::any()); + } + + // Let `size_align` and `size_phase` denote the components stored in + // `size_rounding_align_and_phase`. The normalized layout formula, + // evaluated over the nonnegative integers, is: + // + // object_size(trailing_bytes) + // = self.size_base + // + round_up(size_phase + trailing_bytes, size_align) + // + // Here, `round_up` gives the least multiple of `size_align` greater + // than or equal to its input. Thus `object_size` is nondecreasing. + // By the capacity helper's contract, `max_trailing_bytes` is the + // largest byte count for which `object_size` fits in a `usize`. + // Monotonicity then gives the equivalence: + // + // object_size(trailing_bytes) <= usize::MAX + // iff trailing_bytes <= max_trailing_bytes. + // + // If the helper returns `None`, even `object_size(0)` exceeds the + // limit, so no element count can produce a representable size. + let max_trailing_bytes = match self.max_trailing_bytes(usize::MAX) { + Some(bytes) => bytes, + None => return None, + }; + + // Each of the `elems` elements contributes `self.elem_size` bytes, + // so substitution into the formula requires their product. Checked + // multiplication yields that exact product, including zero for + // zero-sized elements. If it overflows, the complete size cannot fit + // either: `object_size(trailing_bytes) >= trailing_bytes` because + // the base, phase, and rounding increment are all nonnegative. + let trailing_size = match self.elem_size.checked_mul(elems) { + Some(bytes) => bytes, + None => return None, + }; + + // The equivalence above rejects exactly the remaining cases whose + // complete size exceeds `usize::MAX`. After this check we have + // `object_size(trailing_size) <= usize::MAX`. + if trailing_size > max_trailing_bytes { + return None; + } + + // The trailing slice starts at `self.offset` and occupies + // `trailing_size` bytes. Thus `trailing_end` is its end offset modulo + // `usize::MAX + 1`. This sum need not fit: a raw layout need not + // describe a Rust type, so its trailing slice need not lie within + // the complete object size bounded above. + let trailing_end = self.offset.wrapping_add(trailing_size); + + // `padding_for_elems` returns `object_size(trailing_size)` minus the + // trailing-slice end, modulo `usize::MAX + 1`. Adding the end back + // therefore gives `object_size(trailing_size)` modulo the same + // modulus. The bound above establishes that this size fits in + // `usize`, so its residue is the exact size specified by the layout. + let size = trailing_end.wrapping_add(self.padding_for_elems(elems)); + Some(size) + } + + /// Returns `true` only when `self` and `other` describe the same size for + /// every trailing slice length. + /// + /// This recognizes sufficient conditions for equality. A `false` result + /// also includes equivalent size sequences that this test cannot recognize. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&same| { + let (self_align, self_phase) = self.size_rounding_align_and_phase.components(); + let (other_align, other_phase) = other.size_rounding_align_and_phase.components(); + let initial_size = proofs::slice_dst_size_for_trailing_bytes(self, 0); + same == (self.elem_size == other.elem_size + && initial_size.is_some() + && initial_size == proofs::slice_dst_size_for_trailing_bytes(other, 0) + && ((self.elem_size % self_align.get() == 0 + && self.elem_size % other_align.get() == 0) + || (self_align == other_align && self_phase == other_phase))) + }))] + pub(crate) const fn has_same_size_sequence(self, other: Self) -> bool { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::has_same_size_sequence)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().has_same_size_sequence(kani::any()); + } + + // For either layout, interpret its normalized size formula over the + // nonnegative integers: + // + // object_size(elems) + // = size_base + // + round_up(size_phase + elems * elem_size, size_align). + // + // In two cases below, we establish equality of these mathematical sizes + // for every `elems`. Equal sizes also have the same `usize` + // representability bound. + // + // Both cases require a common byte contribution at each element count. + // Equal element sizes give that shared term, `elems * elem_size`. + if self.elem_size != other.elem_size { + return false; + } + + // At `elems == 0`, the element contribution vanishes. Require equal, + // representable initial sizes; each case below then shows that + // increasing `elems` adds the same amount to both initial sizes. + match (self.size_for_elems(0), other.size_for_elems(0)) { + (Some(self_size), Some(other_size)) if self_size == other_size => {} + _ => return false, + } + + let (self_align, self_phase) = self.size_rounding_align_and_phase.components(); + let (other_align, other_phase) = other.size_rounding_align_and_phase.components(); + let self_align = self_align.get(); + let other_align = other_align.get(); + + // Case 1: The element size is a multiple of both alignments. + // + // For two power-of-two alignments, the larger is divisible by the + // smaller. Thus `max_align` is their least common multiple, and a byte + // count is divisible by both alignments exactly when it is divisible by + // `max_align`. + let max_align = if self_align > other_align { self_align } else { other_align }; + + // If the element size is divisible by both alignments, so is `elems * + // elem_size`. Adding an alignment multiple commutes with rounding, + // giving the following identity for each layout: + // + // round_up(size_phase + elems * elem_size, size_align) + // = round_up(size_phase, size_align) + elems * elem_size. + // + // Consequently, `object_size(elems) = object_size(0) + elems * + // elem_size`. The initial sizes and element sizes already agree, so the + // complete size sequences agree. This also covers zero-sized elements, + // whose contribution is zero for every element count. + #[allow(clippy::arithmetic_side_effects)] + if self.elem_size % max_align == 0 { + return true; + } + + // Case 2: The rounding alignments and phases match. + // + // Require a common rounding alignment before comparing phases. If the + // alignments differ, neither case establishes equality. + if self_align != other_align { + return false; + } + + // If the phases also agree, both `round_up` operations have the same + // input and alignment at every element count. In particular, their + // rounded terms agree at zero. Subtracting that common term from the + // equal initial sizes proves `self.size_base == other.size_base`. Both + // the fixed and rounded contributions therefore agree for every + // `elems`, which establishes equality of the complete size formulas. + self_phase == other_phase + } + + /// Advances the size formula by `bytes` while replacing its element size. + /// + /// Let `(size_align, size_phase)` be decoded from + /// `self.size_rounding_align_and_phase`. For a trailing slice length + /// `elems`, the returned layout describes: + /// + /// ```text + /// self.size_base + /// + round_up(size_phase + bytes + elems * elem_size, size_align) + /// ``` + /// + /// Here, `elem_size` is the replacement element size passed to this + /// method. The returned layout preserves the physical slice offset + /// `self.offset` and the rounding alignment; advancing changes the size + /// calculation. + /// + /// Returns `None` if advancing cannot produce representable normalized + /// components. Even when those components fit, evaluating the rounded + /// object size may overflow for some or all trailing slice lengths. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|result| { + let expected_offset = self.size_offset().checked_add(bytes); + match result { + Some(advanced) => { + let align = self.size_rounding_align_and_phase.align(); + expected_offset == Some(advanced.size_offset()) + && advanced.offset == self.offset + && advanced.elem_size == elem_size + && advanced.size_rounding_align_and_phase.align() == align + && (advanced.size_base & (align.get() - 1)) == (self.size_base & (align.get() - 1)) + } + None => expected_offset.is_none(), + } + }))] + const fn advance(self, bytes: usize, elem_size: usize) -> Option { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(TrailingSliceLayout::advance)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().advance(kani::any(), kani::any()); + } + + let (size_align, size_phase) = self.size_rounding_align_and_phase.components(); + + // Advancing replaces the phase inside `round_up` with `size_phase + + // bytes`. To normalize this sum, separate it into whole alignment units + // and a remainder. The whole units can move outside `round_up` into the + // base; the remainder becomes the new phase. + // + // Since `size_align` is a nonzero power of two, `align_mask` selects + // exactly the bits representing a remainder modulo `size_align`. + // Nonzero `size_align` also makes this subtraction representable. + #[allow(clippy::arithmetic_side_effects)] + let align_mask = size_align.get() - 1; + + // The base can increase by at most `usize::MAX - self.size_base`. Since + // only whole alignment units move into the base, the largest + // transferable contribution is: + // + // whole_capacity + // = round_down(usize::MAX - self.size_base, size_align). + // + // A remainder up to `align_mask` stays in the phase and consumes no + // base capacity. Thus the largest advanced phase that can be split into + // representable normalized components is `phase_capacity = + // whole_capacity + align_mask`. Setting the low bits of the remaining + // base capacity computes exactly that sum. Its two terms have disjoint + // set bits, so it cannot overflow. Any larger sum would transfer more + // whole units than the base can hold. + #[allow(clippy::arithmetic_side_effects)] + let phase_capacity = (usize::MAX - self.size_base) | align_mask; + + // The existing phase already occupies `size_phase` bytes of this + // capacity. Subtracting it gives the greatest permitted increment: + // + // bytes <= max_advance + // iff size_phase + bytes <= phase_capacity. + // + // Since `phase_capacity >= align_mask >= size_phase`, the subtraction + // cannot underflow. + #[allow(clippy::arithmetic_side_effects)] + let max_advance = phase_capacity - size_phase; + if bytes > max_advance { + return None; + } + + // This is the fixed term inside rounding after the requested + // advancement, before normalization. The check above proves `size_phase + // + bytes <= phase_capacity <= usize::MAX`, so the addition is + // representable. + #[allow(clippy::arithmetic_side_effects)] + let advanced_phase = size_phase + bytes; + + // Taking the remainder retains precisely the part that cannot move + // outside rounding as whole alignment units. In particular, `0 <= + // normalized_phase < size_align`, as required for the new rounding + // parameters. + let normalized_phase = advanced_phase & align_mask; + + // Rounding down removes precisely the remainder retained in the + // phase, so `whole_bytes = advanced_phase - normalized_phase`. + let whole_bytes = + util::round_down_to_next_multiple_of_alignment(advanced_phase, size_align); + + // Because `advanced_phase <= phase_capacity`, we have + // `whole_bytes <= whole_capacity <= usize::MAX - self.size_base`. + // Adding this transferred contribution to the base therefore fits. + #[allow(clippy::arithmetic_side_effects)] + let size_base = self.size_base + whole_bytes; + + // To verify the returned formula, let `trailing_size = elems * + // elem_size` and evaluate the following over the nonnegative integers. + // Since `whole_bytes` is a multiple of `size_align`, it can move back + // inside `round_up` without changing the result: + // + // size_base + round_up(normalized_phase + trailing_size, size_align) + // = self.size_base + // + round_up(whole_bytes + normalized_phase + trailing_size, size_align) + // = self.size_base + // + round_up(size_phase + bytes + trailing_size, size_align). + // + // This is the requested advanced formula. The replacement element size + // supplies its new coefficient of `elems`; `offset` retains the + // caller's field-placement description. + Some(Self { + offset: self.offset, + elem_size, + size_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(size_align, normalized_phase), + }) + } } impl SizeInfo { @@ -73,12 +725,42 @@ impl SizeInfo { #[allow(unused)] #[cfg_attr(not(zerocopy_inline_always), inline)] #[cfg_attr(zerocopy_inline_always, inline(always))] + #[cfg_attr(kani, kani::ensures(|result| match (*self, *result) { + (SizeInfo::Sized { size }, Some(SizeInfo::Sized { size: converted })) => { + size == converted + } + (SizeInfo::SliceDst(original), Some(SizeInfo::SliceDst(converted))) => { + converted.elem_size.get() == original.elem_size + && converted.offset == original.offset + && converted.size_base == original.size_base + && converted.size_rounding_align_and_phase == original.size_rounding_align_and_phase + } + (SizeInfo::SliceDst(original), None) => original.elem_size == 0, + _ => false, + }))] const fn try_to_nonzero_elem_size(&self) -> Option> { + #[cfg(kani)] + #[kani::proof_for_contract(SizeInfo::try_to_nonzero_elem_size)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().try_to_nonzero_elem_size(); + } + Some(match *self { SizeInfo::Sized { size } => SizeInfo::Sized { size }, - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => { + SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base, + size_rounding_align_and_phase, + }) => { if let Some(elem_size) = NonZeroUsize::new(elem_size) { - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) + SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base, + size_rounding_align_and_phase, + }) } else { return None; } @@ -87,6 +769,40 @@ impl SizeInfo { } } +/// Returns the largest number of `elem_size`-byte elements which fit in +/// `bytes`, along with the number of bytes those elements occupy. +#[cfg_attr( + kani, + kani::ensures(|&(elems, used)| { + (match elems.checked_mul(elem_size.get()) { + Some(product) => { + product == used && used <= bytes && bytes - used < elem_size.get() + } + None => false, + }) && used + .checked_add(elem_size.get()) + .map(|next_used| next_used > bytes) + .unwrap_or(true) + }) +)] +#[inline(always)] +const fn max_elems_for_bytes(bytes: usize, elem_size: NonZeroUsize) -> (usize, usize) { + #[cfg(kani)] + #[kani::proof_for_contract(max_elems_for_bytes)] + #[kani::solver(kissat)] + fn proof() { + max_elems_for_bytes(kani::any(), kani::any()); + } + + #[allow(clippy::arithmetic_side_effects)] + let elems = bytes / elem_size.get(); + let used = match elems.checked_mul(elem_size.get()) { + Some(used) => used, + None => const_unreachable!(), + }; + (elems, used) +} + #[doc(hidden)] #[derive(Copy, Clone)] #[cfg_attr(test, derive(Debug))] @@ -97,6 +813,7 @@ pub enum CastType { } #[cfg_attr(test, derive(Debug))] +#[cfg_attr(kani, derive(kani::Arbitrary))] pub(crate) enum MetadataCastError { Alignment, Size, @@ -128,7 +845,7 @@ impl DstLayout { /// 229. #[cfg(not(kani))] #[cfg(not(target_pointer_width = "16"))] - pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 28) { + pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 29) { Some(max_align) => max_align, None => const_unreachable!(), }; @@ -164,7 +881,18 @@ impl DstLayout { /// shallow padding, unsafe code may assume that the result of this method /// accurately reflects the size, alignment, and lack of static shallow /// padding of that type. + #[cfg_attr(kani, kani::ensures(|result| { + result.align == self.align && result.size_info == self.size_info + && result.statically_shallow_unpadded + }))] const fn assume_shallow_unpadded(self) -> Self { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::assume_shallow_unpadded)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().assume_shallow_unpadded(); + } + Self { statically_shallow_unpadded: true, ..self } } @@ -182,7 +910,20 @@ impl DstLayout { #[doc(hidden)] #[must_use] #[inline] + #[cfg_attr(kani, kani::requires(repr_align.map_or(true, |align| align.is_power_of_two())))] + #[cfg_attr(kani, kani::ensures(|result| { + result.align == repr_align.unwrap_or(Self::MIN_ALIGN) + && result.size_info == SizeInfo::Sized { size: 0 } + && result.statically_shallow_unpadded + }))] pub const fn new_zst(repr_align: Option) -> DstLayout { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::new_zst)] + #[kani::solver(kissat)] + fn proof() { + let _ = DstLayout::new_zst(kani::any()); + } + let align = match repr_align { Some(align) => align, None => Self::MIN_ALIGN, @@ -206,7 +947,24 @@ impl DstLayout { #[doc(hidden)] #[must_use] #[inline] + #[cfg_attr(kani, kani::ensures(|result| { + result.align.get() == mem::align_of::() + && result.size_info == SizeInfo::Sized { size: mem::size_of::() } + && result.statically_shallow_unpadded == false + }))] pub const fn for_type() -> DstLayout { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_type::<()>)] + fn proof_zst() { + let _ = DstLayout::for_type::<()>(); + } + + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_type::<(u8, u64)>)] + fn proof_padded() { + let _ = DstLayout::for_type::<(u8, u64)>(); + } + // SAFETY: `align` is correct by construction. `T: Sized`, and so it is // sound to initialize `size_info` to `SizeInfo::Sized { size }`; the // `size` field is also correct by construction. `unpadded` can safely @@ -221,16 +979,39 @@ impl DstLayout { } } - /// Constructs a `DstLayout` which describes a `T` that does not contain - /// padding. + /// Produces `T`'s size and alignment with static shallow padding assumed + /// absent. + /// + /// This method records the assumption without checking `T` for padding. + /// It can be used to ignore padding within a field when computing padding + /// introduced by its enclosing struct. /// /// # Safety /// - /// Unsafe code may assume that `DstLayout` is the correct layout for `T`. + /// Unsafe code may rely on the result's size and alignment being correct + /// for `T`. Relying on its recorded absence of static shallow padding + /// requires an independent justification that `T` lacks such padding. #[doc(hidden)] #[must_use] #[inline] + #[cfg_attr(kani, kani::ensures(|result| { + result.align.get() == mem::align_of::() + && result.size_info == SizeInfo::Sized { size: mem::size_of::() } + && result.statically_shallow_unpadded == true + }))] pub const fn for_unpadded_type() -> DstLayout { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_unpadded_type::<()>)] + fn proof_zst() { + let _ = DstLayout::for_unpadded_type::<()>(); + } + + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_unpadded_type::<(u8, u64)>)] + fn proof_padded() { + let _ = DstLayout::for_unpadded_type::<(u8, u64)>(); + } + Self::for_type::().assume_shallow_unpadded() } @@ -239,7 +1020,31 @@ impl DstLayout { /// # Safety /// /// Unsafe code may assume that `DstLayout` is the correct layout for `[T]`. + #[cfg_attr(kani, kani::ensures(|result| { + result.align.get() == mem::align_of::() && result.statically_shallow_unpadded + && matches!(result.size_info, SizeInfo::SliceDst(trailing) + if trailing.offset == 0 && trailing.size_base == 0 + && trailing.elem_size == mem::size_of::() + && trailing.size_rounding_align_and_phase.components() == (result.align, 0)) + }))] pub(crate) const fn for_slice() -> DstLayout { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_slice::<()>)] + fn proof_zst() { + let _ = DstLayout::for_slice::<()>(); + } + + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::for_slice::<(u8, u64)>)] + fn proof_padded() { + let _ = DstLayout::for_slice::<(u8, u64)>(); + } + + let align = match NonZeroUsize::new(mem::align_of::()) { + Some(align) => align, + None => const_unreachable!(), + }; + // SAFETY: The alignment of a slice is equal to the alignment of its // element type, and so `align` is initialized correctly. // @@ -249,13 +1054,12 @@ impl DstLayout { // construction. Since `[T]` is a (degenerate case of a) slice DST, it // is correct to initialize `size_info` to `SizeInfo::SliceDst`. DstLayout { - align: match NonZeroUsize::new(mem::align_of::()) { - Some(align) => align, - None => const_unreachable!(), - }, + align, size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset: 0, elem_size: mem::size_of::(), + size_base: 0, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, 0), }), statically_shallow_unpadded: true, } @@ -273,15 +1077,42 @@ impl DstLayout { /// alignment modifiers and fields corresponding to that definition, the /// resulting `DstLayout` will correctly encode the layout of that struct. /// + /// This guarantee uses one explicit compatibility premise for generic + /// indirection: while Rust accepts an instantiation which transitively + /// places a `repr(align)` type inside a `repr(packed)` type, that + /// instantiation follows the otherwise-applicable `repr(C)`, + /// `repr(packed)`, and inner-type layout rules. The Reference prohibits the + /// direct form, and Rust's continued acceptance and layout behavior are + /// outside the algebra proved by this implementation. Re-audit if the + /// compiler rejects such an instantiation or assigns it a different + /// layout. + /// /// We make no guarantees to the behavior of this method when it is invoked /// with arguments that cannot correspond to a valid `repr(C)` struct. #[must_use] #[inline] + #[cfg_attr(kani, kani::requires(proofs::repr_c_layout(repr_align, repr_packed, fields).is_some()))] + #[cfg_attr(kani, kani::ensures(|&result| { + Some(result) == proofs::repr_c_layout(repr_align, repr_packed, fields) + }))] pub const fn for_repr_c_struct( repr_align: Option, repr_packed: Option, fields: &[DstLayout], ) -> DstLayout { + // This harness covers up to three fields, including an optional final + // DST. Arbitrary-length composition is not established by this bound. + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(DstLayout::for_repr_c_struct)] + #[kani::solver(kissat)] + #[kani::unwind(4)] + fn proof() { + let fields: [DstLayout; 3] = kani::any(); + let len: usize = kani::any(); + kani::assume(len <= fields.len()); + let _ = DstLayout::for_repr_c_struct(kani::any(), kani::any(), &fields[..len]); + } + let mut layout = DstLayout::new_zst(repr_align); let mut i = 0; @@ -338,7 +1169,18 @@ impl DstLayout { #[doc(hidden)] #[must_use] #[inline] + #[cfg_attr(kani, kani::requires(proofs::extended_layout(self, field, repr_packed).is_some()))] + #[cfg_attr(kani, kani::ensures(|&result| { + Some(result) == proofs::extended_layout(self, field, repr_packed) + }))] pub const fn extend(self, field: DstLayout, repr_packed: Option) -> Self { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::extend)] + #[kani::solver(kissat)] + fn proof() { + let _ = kani::any::().extend(kani::any(), kani::any()); + } + use util::{max, min, padding_needed_for}; // If `repr_packed` is `None`, there are no alignment constraints, and @@ -427,6 +1269,8 @@ impl DstLayout { SizeInfo::SliceDst(TrailingSliceLayout { offset: trailing_offset, elem_size, + size_base, + size_rounding_align_and_phase, }) => { // If the trailing field is dynamically sized, so too // will the resulting layout. The offset of the trailing @@ -440,11 +1284,30 @@ impl DstLayout { // panic otherwise (e.g., combining or aligning the // components would create a size exceeding // `usize::MAX`). - let offset = match offset.checked_add(trailing_offset) { + let trailing_offset = match offset.checked_add(trailing_offset) { Some(offset) => offset, None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"), }; - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) + + // Before normalization, the composite's size is: + // + // offset + size_base + // + round_up(size_phase + elems * elem_size, size_align) + // + // `offset` does not participate in the rounding, + // so it can be added directly to the normalized + // base. + let size_base = match offset.checked_add(size_base) { + Some(base) => base, + None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"), + }; + + SizeInfo::SliceDst(TrailingSliceLayout { + offset: trailing_offset, + elem_size, + size_base, + size_rounding_align_and_phase, + }) } }, ) @@ -459,18 +1322,11 @@ impl DstLayout { } /// Like `Layout::pad_to_align`, this routine rounds the size of this layout - /// up to the nearest multiple of this type's alignment or `repr_packed` - /// (whichever is less). This method leaves DST layouts unchanged, since the - /// trailing padding of DSTs is computed at runtime. - /// - /// The accompanying boolean is `true` if the resulting composition of - /// fields necessitated static (as opposed to dynamic) padding; otherwise - /// `false`. + /// up to the nearest multiple of this type's alignment. For DST layouts, + /// this updates the runtime size formula to account for trailing padding. /// /// In order to match the layout of a `#[repr(C)]` struct, this method - /// should be invoked after the invocations of [`DstLayout::extend`]. If - /// `self` corresponds to a type marked with `repr(packed(N))`, then - /// `repr_packed` should be set to `Some(N)`, otherwise `None`. + /// should be invoked after the invocations of [`DstLayout::extend`]. /// /// This method cannot be used to match the layout of a record with the /// default representation, as that representation is mostly unspecified. @@ -480,7 +1336,7 @@ impl DstLayout { /// If a (potentially hypothetical) valid `repr(C)` type begins with fields /// whose layout are `self` followed only by zero or more bytes of trailing /// padding (not included in `self`), then unsafe code may rely on - /// `self.pad_to_align(repr_packed)` producing a layout that correctly + /// `self.pad_to_align()` producing a layout that correctly /// encapsulates the layout of that type. /// /// We make no guarantees to the behavior of this method if `self` cannot @@ -489,7 +1345,16 @@ impl DstLayout { #[doc(hidden)] #[must_use] #[inline] + #[cfg_attr(kani, kani::requires(proofs::padded_layout(self).is_some()))] + #[cfg_attr(kani, kani::ensures(|&result| Some(result) == proofs::padded_layout(self)))] pub const fn pad_to_align(self) -> Self { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(DstLayout::pad_to_align)] + #[kani::solver(kissat)] + fn proof() { + let _ = kani::any::().pad_to_align(); + } + use util::padding_needed_for; let (static_padding, size_info) = match self.size_info { @@ -504,10 +1369,69 @@ impl DstLayout { (padding, SizeInfo::Sized { size }) } // For DST layouts, trailing padding depends on the length of the - // trailing DST and is computed at runtime. This does not alter the - // offset or element size of the layout, so we leave `size_info` - // unchanged. - size_info @ SizeInfo::SliceDst(_) => (0, size_info), + // trailing DST and is computed at runtime. Normalize the composed + // rounding operation back into the representation documented on + // `TrailingSliceLayout`. + SizeInfo::SliceDst(mut trailing) => { + let (size_align, size_phase) = trailing.size_rounding_align_and_phase.components(); + if size_align.get() < self.align.get() { + // For a trailing-slice byte count `trailing_size`, the + // input layout describes: + // + // trailing.size_base + // + round_up(size_phase + trailing_size, size_align) + // + // Here, `size_align < self.align`. Both alignments are + // powers of two, so rounding the complete input size to + // `self.align` is equivalent to: + // + // rounded_base = round_up(trailing.size_base, size_align) + // fixed_contribution = rounded_base + size_phase + // new_phase = fixed_contribution % self.align + // new_base = fixed_contribution - new_phase + // new_base + round_up(new_phase + trailing_size, self.align) + // + // `prove_dst_layout_pad_to_align_dst_outer_alignment_transition` + // checks the exact field transition and preserved + // invariants. Its `_formula` companion proves this + // identity for every representable trailing byte count. + let base_padding = padding_needed_for(trailing.size_base, size_align); + let rounded_base = match trailing.size_base.checked_add(base_padding) { + Some(base) => base, + None => const_panic!("Adding padding caused size to overflow `usize`."), + }; + let fixed_contribution = match rounded_base.checked_add(size_phase) { + Some(bytes) => bytes, + None => const_panic!("Adding padding caused size to overflow `usize`."), + }; + #[allow(clippy::arithmetic_side_effects)] + let phase = fixed_contribution & (self.align.get() - 1); + // Rounding down removes `phase`, the fixed contribution + // modulo `self.align`, leaving the new base. + let size_base = util::round_down_to_next_multiple_of_alignment( + fixed_contribution, + self.align, + ); + trailing.size_base = size_base; + trailing.size_rounding_align_and_phase = + RoundingAlignAndPhase::new(self.align, phase); + } else { + // `size_align` is a multiple of `self.align`, so + // the inner rounded size is already aligned to + // `self.align`. Only `size_base` needs to be rounded. + // `prove_dst_layout_pad_to_align_dst_inner_rounding_transition` + // checks the exact field transition and preserved + // invariants. Its `_formula` companion proves size + // equivalence for every representable trailing byte count. + let padding = padding_needed_for(trailing.size_base, self.align); + trailing.size_base = match trailing.size_base.checked_add(padding) { + Some(base) => base, + None => const_panic!("Adding padding caused size to overflow `usize`."), + }; + } + + (0, SizeInfo::SliceDst(trailing)) + } }; let statically_shallow_unpadded = self.statically_shallow_unpadded && static_padding == 0; @@ -515,48 +1439,127 @@ impl DstLayout { DstLayout { align: self.align, size_info, statically_shallow_unpadded } } - /// Produces `true` if `self` requires static padding; otherwise `false`. + /// Produces `false` when this layout is marked as having no static shallow + /// padding, and `true` otherwise. + /// + /// A `true` result is conservative: it does not establish that the type + /// contains padding. For example, this method produces `true` for + /// `DstLayout::for_type::()`, since `for_type` does not record whether + /// static shallow padding is absent. #[must_use] #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&padding| padding == !self.statically_shallow_unpadded))] pub const fn requires_static_padding(self) -> bool { + #[cfg(kani)] + #[kani::proof_for_contract(DstLayout::requires_static_padding)] + #[kani::solver(kissat)] + fn proof() { + let _ = kani::any::().requires_static_padding(); + } + !self.statically_shallow_unpadded } - /// Produces `true` if there exists any metadata for which a type of layout - /// `self` would require dynamic trailing padding; otherwise `false`. + /// Produces `false` only if every valid metadata describes an instance + /// which requires no dynamic trailing padding. + /// + /// A `true` result conservatively selects the dynamic-padding path; it does + /// not guarantee that any valid metadata actually requires padding. #[must_use] #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&padding| match self.size_info { + SizeInfo::Sized { .. } => !padding, + SizeInfo::SliceDst(trailing) => padding == ( + proofs::slice_dst_size_for_trailing_bytes(trailing, 0) != Some(trailing.offset) + || trailing.elem_size % trailing.size_rounding_align_and_phase.align().get() != 0 + ), + }))] pub const fn requires_dynamic_padding(self) -> bool { - // A `% self.align.get()` cannot panic, since `align` is non-zero. - #[allow(clippy::arithmetic_side_effects)] + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(DstLayout::requires_dynamic_padding)] + #[kani::solver(kissat)] + fn proof() { + let _ = kani::any::().requires_dynamic_padding(); + } + match self.size_info { SizeInfo::Sized { .. } => false, SizeInfo::SliceDst(trailing_slice_layout) => { - // SAFETY: This predicate is formally proved sound by - // `proofs::prove_requires_dynamic_padding`. - trailing_slice_layout.offset % self.align.get() != 0 - || trailing_slice_layout.elem_size % self.align.get() != 0 + // SAFETY: `proofs::prove_requires_dynamic_padding` formally + // proves the safety-critical direction: when this predicate + // returns false, no valid metadata requires dynamic padding. + match trailing_slice_layout.size_for_elems(0) { + Some(initial_size) => { + #[allow(clippy::arithmetic_side_effects)] + let has_static_stride = trailing_slice_layout.elem_size + % trailing_slice_layout.size_rounding_align_and_phase.align().get() + == 0; + initial_size != trailing_slice_layout.offset || !has_static_stride + } + None => true, + } + } + } + } + + /// Returns the largest trailing slice length whose object size is exactly + /// `size`. + /// + /// Returns `None` if there is no such length, if this describes a sized + /// type, or if the trailing slice element is zero-sized. + #[inline(always)] + #[cfg_attr(kani, kani::ensures(|&result| { + result == match self.size_info { + SizeInfo::SliceDst(trailing) if trailing.elem_size != 0 => { + proofs::max_trailing_bytes(trailing, size) + .map(|bytes| bytes / trailing.elem_size) + .filter(|&elems| trailing.size_for_elems(elems) == Some(size)) + } + _ => None, + } + }))] + const fn metadata_for_exact_size(&self, size: usize) -> Option { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(DstLayout::metadata_for_exact_size)] + #[kani::solver(kissat)] + fn proof() { + kani::any::().metadata_for_exact_size(kani::any()); + } + + match self.size_info { + SizeInfo::Sized { .. } + | SizeInfo::SliceDst(TrailingSliceLayout { elem_size: 0, .. }) => None, + SizeInfo::SliceDst(_) => { + // Address zero is aligned to every supported alignment, and + // `0 + size` cannot overflow. The validator returns the + // largest metadata whose object fits in `size`; accepting it + // only when the resulting object consumes all `size` bytes + // turns that bounded query into an exact-size query. + match self.validate_cast_and_convert_metadata(0, size, CastType::Prefix) { + Ok((elems, object_size)) if object_size == size => Some(elems), + _ => None, + } } } } - /// Validates that a cast is sound from a layout perspective. + /// Given a completed type layout, produces metadata and a split index for + /// a prefix or suffix cast that satisfies the type's size and alignment + /// requirements. /// - /// Validates that the size and alignment requirements of a type with the - /// layout described in `self` would not be violated by performing a - /// `cast_type` cast from a pointer with address `addr` which refers to a - /// memory region of size `bytes_len`. + /// `addr` and `bytes_len` describe the address and size of the source + /// memory region, and `cast_type` selects its prefix or suffix. /// /// If the cast is valid, `validate_cast_and_convert_metadata` returns - /// `(elems, split_at)`. If `self` describes a dynamically-sized type, then - /// `elems` is the maximum number of trailing slice elements for which a - /// cast would be valid (for sized types, `elem` is meaningless and should - /// be ignored). `split_at` is the index at which to split the memory region - /// in order for the prefix (suffix) to contain the result of the cast, and - /// in order for the remaining suffix (prefix) to contain the leftover - /// bytes. + /// `Ok((elems, split_at))`. For a dynamically-sized type, `elems` is the + /// maximum number of trailing slice elements for which the cast is valid. + /// For sized types, `elems` is meaningless and should be ignored. + /// `split_at` is the index at which to split the memory region so that the + /// prefix (suffix) contains the result of the cast and the remaining + /// suffix (prefix) contains the leftover bytes. /// /// There are three conditions under which a cast can fail: + /// /// - The smallest possible value for the type is larger than the provided /// memory region /// - A prefix cast is requested, and `addr` does not satisfy `self`'s @@ -568,10 +1571,10 @@ impl DstLayout { /// /// # Safety /// - /// The caller may assume that this implementation is correct, and may rely - /// on that assumption for the soundness of their code. In particular, the - /// caller may assume that, if `validate_cast_and_convert_metadata` returns - /// `Some((elems, split_at))`, then: + /// When `self` accurately describes a completed type layout, including + /// trailing padding, and `addr + bytes_len` fits in `usize`, callers may + /// rely on the following guarantees for `Ok((elems, split_at))`: + /// /// - A pointer to the type (for dynamically sized types, this includes /// `elems` as its pointer metadata) describes an object of size `size <= /// bytes_len` @@ -582,6 +1585,11 @@ impl DstLayout { /// - `split_at == bytes_len - size` /// - `addr + split_at` satisfies `self`'s alignment /// + /// These guarantees require the completed-layout premise; `DstLayout` + /// also permits unfinished layout fragments and arbitrary field values. + /// In particular, suffix alignment relies on every object size being a + /// multiple of the type's alignment. + /// /// Note that this method does *not* ensure that a pointer constructed from /// its return values will be a valid pointer. In particular, this method /// does not reason about `isize` overflow, which is a requirement of many @@ -606,12 +1614,57 @@ impl DstLayout { /// condition, even if `debug_assertions` are enabled. #[allow(unused)] #[inline(always)] + #[cfg_attr(kani, kani::requires(addr.checked_add(bytes_len).is_some()))] + #[cfg_attr(kani, kani::requires(!matches!(self.size_info, + SizeInfo::SliceDst(TrailingSliceLayout { elem_size: 0, .. }))))] + #[cfg_attr(kani, kani::ensures(|result| { + let endpoint = match cast_type { + CastType::Prefix => addr, + CastType::Suffix => addr + bytes_len, + }; + if endpoint % self.align.get() != 0 { + return matches!(result, Err(MetadataCastError::Alignment)); + } + let candidate = match self.size_info { + SizeInfo::Sized { size } => { + if size <= bytes_len { Some((0, size)) } else { None } + } + SizeInfo::SliceDst(trailing) => { + proofs::max_trailing_bytes(trailing, bytes_len).and_then(|bytes| { + let elems = bytes / trailing.elem_size; + trailing.size_for_elems(elems).map(|size| (elems, size)) + }) + } + }; + match (result, candidate) { + (Ok((elems, split_at)), Some((expected_elems, size))) => { + *elems == expected_elems && *split_at == match cast_type { + CastType::Prefix => size, + CastType::Suffix => bytes_len - size, + } + } + (Err(MetadataCastError::Size), None) => true, + _ => false, + } + }))] pub(crate) const fn validate_cast_and_convert_metadata( &self, addr: usize, bytes_len: usize, cast_type: CastType, ) -> Result<(usize, usize), MetadataCastError> { + #[cfg(all(kani, kani_slow))] + #[kani::proof_for_contract(DstLayout::validate_cast_and_convert_metadata)] + #[kani::solver(kissat)] + fn proof() { + let cast_type = if kani::any() { CastType::Prefix } else { CastType::Suffix }; + let _ = kani::any::().validate_cast_and_convert_metadata( + kani::any(), + kani::any(), + cast_type, + ); + } + // `debug_assert!`, but with `#[allow(clippy::arithmetic_side_effects)]`. macro_rules! __const_debug_assert { ($e:expr $(, $msg:expr)?) => { @@ -651,9 +1704,8 @@ impl DstLayout { // of this check is trivial - `addr` is the address the object will // live at. // - // For a suffix cast, we know that all valid sizes for the type are - // a multiple of the alignment (and by safety precondition, we know - // `DstLayout` may only describe valid Rust types). Thus, a + // Under the documented completed-layout premise, all valid sizes + // for the type are multiples of its alignment. Thus, a // validly-sized instance which lives at a validly-aligned address // must also end at a validly-aligned address. Thus, if the end // address for a suffix cast (`addr + bytes_len`) is not aligned, @@ -666,7 +1718,7 @@ impl DstLayout { // Addition is guaranteed not to overflow because `offset <= // bytes_len`, and `addr + bytes_len <= usize::MAX` is a // precondition of this method. Modulus is guaranteed not to divide - // by 0 because `align` is non-zero. + // by zero because `self.align` is non-zero. #[allow(clippy::arithmetic_side_effects)] if (addr + offset) % self.align.get() != 0 { return Err(MetadataCastError::Alignment); @@ -680,59 +1732,50 @@ impl DstLayout { } (0, size) } - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => { - // Calculate the maximum number of bytes that could be consumed - // - any number of bytes larger than this will either not be a - // multiple of the alignment, or will be larger than - // `bytes_len`. - let max_total_bytes = - util::round_down_to_next_multiple_of_alignment(bytes_len, self.align); - // Calculate the maximum number of bytes that could be consumed - // by the trailing slice. - // - // FIXME(#67): Once our MSRV is 1.65, use let-else: - // https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#let-else-statements - let max_slice_and_padding_bytes = match max_total_bytes.checked_sub(offset) { - Some(max) => max, - // `bytes_len` too small even for 0 trailing slice elements. + SizeInfo::SliceDst(trailing) => { + let max_slice_bytes = match trailing.max_trailing_bytes(bytes_len) { + Some(bytes) => bytes, None => return Err(MetadataCastError::Size), }; - // Calculate the number of elements that fit in - // `max_slice_and_padding_bytes`; any remaining bytes will be - // considered padding. + // `max_slice_bytes`. Any remaining input bytes stay outside + // the selected object unless the object's own rounding + // operation consumes them as padding. // // Guaranteed not to divide by zero: `elem_size` is non-zero. - #[allow(clippy::arithmetic_side_effects)] - let elems = max_slice_and_padding_bytes / elem_size.get(); - // Guaranteed not to overflow on multiplication: `usize::MAX >= - // max_slice_and_padding_bytes >= (max_slice_and_padding_bytes / - // elem_size) * elem_size`. + let (elems, _) = max_elems_for_bytes(max_slice_bytes, trailing.elem_size); + // The remainder is `max_slice_bytes - elems * elem_size`. + // Writing it as a remainder exposes its bound to the + // optimizer: when `elem_size <= size_align`, rounding the + // remainder down to `size_align` always gives zero. + let size_align = trailing.size_rounding_align_and_phase.align(); + // `size_phase + max_slice_bytes == max_rounded_bytes`, an + // alignment multiple. The selected tail removes `unused_bytes` + // from this boundary; rounding back up removes only whole + // alignment units. Thus its normalized rounded size is: // - // Guaranteed not to overflow on addition: - // - max_slice_and_padding_bytes == max_total_bytes - offset - // - elems * elem_size <= max_slice_and_padding_bytes == max_total_bytes - offset - // - elems * elem_size + offset <= max_total_bytes <= usize::MAX - #[allow(clippy::arithmetic_side_effects)] - let without_padding = offset + elems * elem_size.get(); - // `self_bytes` is equal to the offset bytes plus the bytes - // consumed by the trailing slice plus any padding bytes - // required to satisfy the alignment. Note that we have computed - // the maximum number of trailing slice elements that could fit - // in `self_bytes`, so any padding is guaranteed to be less than - // the size of an extra element. + // round_up(size_phase + elems * elem_size, size_align) + // = max_rounded_bytes - round_down(unused_bytes, size_align) // - // Guaranteed not to overflow: - // - By previous comment: without_padding == elems * elem_size + - // offset <= max_total_bytes - // - By construction, `max_total_bytes` is a multiple of - // `self.align`. - // - At most, adding padding needed to round `without_padding` - // up to the next multiple of the alignment will bring - // `self_bytes` up to `max_total_bytes`. + // `prove_slice_dst_validator_selected_size_fits` checks this + // identity and its intermediate bounds, using the relations + // in `prove_max_trailing_bytes_components`. The capacity is + // exact by `prove_max_trailing_bytes`; the element helper's + // contract establishes the largest whole-element count. #[allow(clippy::arithmetic_side_effects)] - let self_bytes = - without_padding + util::padding_needed_for(without_padding, self.align); + let unused_bytes = max_slice_bytes % trailing.elem_size.get(); + let unused_aligned_bytes = + util::round_down_to_next_multiple_of_alignment(unused_bytes, size_align); + #[allow(clippy::arithmetic_side_effects)] + let max_rounded_bytes = util::round_down_to_next_multiple_of_alignment( + bytes_len - trailing.size_base, + size_align, + ); + // `unused_aligned_bytes <= unused_bytes <= max_slice_bytes + // <= max_rounded_bytes`, and `size_base + max_rounded_bytes + // <= bytes_len`, so neither operation can overflow. + #[allow(clippy::arithmetic_side_effects)] + let self_bytes = trailing.size_base + (max_rounded_bytes - unused_aligned_bytes); (elems, self_bytes) } }; @@ -745,7 +1788,7 @@ impl DstLayout { // - In the `Sized` branch, only returns `size` if `size <= // bytes_len`. // - In the `SliceDst` branch, calculates `self_bytes <= - // max_toatl_bytes`, which is upper-bounded by `bytes_len`. + // bytes_len`. #[allow(clippy::arithmetic_side_effects)] CastType::Suffix => bytes_len - self_bytes, }; @@ -754,6 +1797,95 @@ impl DstLayout { } } +/// Computes the affine destination-metadata map. +/// +/// # Safety +/// +/// `base + metadata * multiple`, evaluated over the mathematical nonnegative +/// integers, must not exceed `usize::MAX`. +#[inline(always)] +#[allow(unstable_name_collisions)] +#[cfg_attr(kani, kani::requires(metadata.checked_mul(multiple) + .and_then(|scaled| base.checked_add(scaled)).is_some()))] +#[cfg_attr(kani, kani::ensures(|&result| { + metadata.checked_mul(multiple).and_then(|scaled| base.checked_add(scaled)) == Some(result) +}))] +unsafe fn add_scaled_metadata(base: usize, metadata: usize, multiple: usize) -> usize { + // The call-site safety argument establishes that both operations are + // representable for valid source metadata. This harness verifies the + // production implementation selected by Kani's modern rustc against + // checked arithmetic under precisely that precondition. The distributive + // natural-number argument which establishes the precondition is kept + // beside the unsafe pointer projection where it can be reviewed with the + // layout premises that it consumes. + #[cfg(kani)] + #[kani::proof_for_contract(add_scaled_metadata)] + #[kani::solver(kissat)] + fn proof() { + let base: usize = kani::any(); + let metadata: usize = kani::any(); + let multiple: usize = kani::any(); + let Some(scaled) = metadata.checked_mul(multiple) else { + kani::assume(false); + loop {} + }; + let Some(expected) = base.checked_add(scaled) else { + kani::assume(false); + loop {} + }; + + // SAFETY: The checked operations above returned `Some`, so the + // operation is representable in `usize`. The standard-library + // contracts say that this rules out undefined behavior for both + // unchecked operations [1][2]. + // + // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_mul: + // + // This results in undefined behavior [..] when `checked_mul` would + // return `None`. + // + // [2] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_add: + // + // This results in undefined behavior [..] when `checked_add` would + // return `None`. + assert_eq!(unsafe { add_scaled_metadata(base, metadata, multiple) }, expected); + } + + #[allow(unused_imports)] + use crate::util::polyfills::*; + + // SAFETY: The caller promises that `base + metadata * multiple` is + // representable over the nonnegative integers. Thus `metadata * multiple` + // is representable too. On toolchains with inherent unchecked arithmetic, + // this is exactly the condition under which `usize::unchecked_mul` does + // not have undefined behavior [1]. On older supported toolchains, method + // resolution selects `polyfills::NumExt::unchecked_mul`, whose safety + // contract requires the same no-overflow condition. Kani checks the two + // implementations in separate harnesses. + // + // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_mul: + // + // This results in undefined behavior when `self * rhs > usize::MAX` or + // `self * rhs < usize::MIN`, i.e. when `checked_mul` would return `None`. + // + let scaled = unsafe { metadata.unchecked_mul(multiple) }; + + // SAFETY: The caller promises that `base + metadata * multiple` is + // representable in `usize`, and `scaled == metadata * multiple`. On + // toolchains with inherent unchecked arithmetic, that is exactly the + // condition under which `usize::unchecked_add` does not have undefined + // behavior [1]. On older supported toolchains, method resolution selects + // `polyfills::NumExt::unchecked_add`, whose safety contract requires the + // same no-overflow condition. Kani checks the two implementations in + // separate harnesses. + // + // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_add: + // + // This results in undefined behavior when `self + rhs > usize::MAX` or + // `self + rhs < usize::MIN`, i.e. when `checked_add` would return `None`. + unsafe { base.unchecked_add(scaled) } +} + pub(crate) use cast_from::CastFrom; mod cast_from { use crate::*; @@ -823,52 +1955,19 @@ mod cast_from { // - If this is possible, any information necessary to perform // the `Src`->`Dst` metadata conversion at runtime. // - // Assume that `Src` and `Dst` are slice DSTs, and define: - // - `S_OFF = Src::LAYOUT.size_info.offset` - // - `S_ELEM = Src::LAYOUT.size_info.elem_size` - // - `D_OFF = Dst::LAYOUT.size_info.offset` - // - `D_ELEM = Dst::LAYOUT.size_info.elem_size` - // - // We are trying to solve the following equation: - // - // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM - // - // At runtime, we will be attempting to compute `d_meta`, given - // `s_meta` (a runtime value) and all other parameters (which - // are compile-time values). We can solve like so: - // - // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM - // - // d_meta * D_ELEM = S_OFF - D_OFF + s_meta * S_ELEM - // - // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM - // - // Since `d_meta` will be a `usize`, we need the right-hand side - // to be an integer, and this needs to hold for *any* value of - // `s_meta` (in order for our conversion to be infallible - ie, - // to not have to reject certain values of `s_meta` at runtime). - // This means that: - // - // - `s_meta * S_ELEM` must be a multiple of `D_ELEM` - // - Since this must hold for any value of `s_meta`, `S_ELEM` - // must be a multiple of `D_ELEM` - // - `S_OFF - D_OFF` must be a multiple of `D_ELEM` - // - // Thus, let `OFFSET_DELTA_ELEMS = (S_OFF - D_OFF)/D_ELEM` and - // `ELEM_MULTIPLE = S_ELEM/D_ELEM`. We can rewrite the above - // expression as: - // - // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM + // For slice DSTs, destination metadata is an affine function + // of source metadata: // - // d_meta = OFFSET_DELTA_ELEMS + s_meta * ELEM_MULTIPLE + // dst_meta = offset_delta_elems + src_meta * elem_multiple // - // Thus, we just need to compute the following and confirm that - // they have integer solutions in order to both a) determine - // whether infallible `Src` -> `Dst` casts are possible and, b) - // pre-compute the parameters necessary to perform those casts - // at runtime. These parameters are encapsulated in - // `CastParams`, which acts as a witness that such infallible - // casts are possible. + // `elem_multiple` scales the destination's trailing element + // size to the source's. `offset_delta_elems` selects a + // destination size equal to the source's zero-element size. + // We then prove that the two complete rounded size formulas + // produce the same sequence for all metadata. This is + // necessary because a packed outer DST can retain a nested + // field's rounding operation; comparing only physical + // trailing-slice offsets is not sufficient. /// The parameters required in order to perform an /// unsized-to-unsized pointer cast from `Src` to `Dst` as /// described above. @@ -910,15 +2009,60 @@ mod cast_from { } impl CastParams { + /// Given a nonzero `dst.elem_size` that exactly divides + /// `src.elem_size`, produces `true` only when the following + /// metadata map preserves the complete object size for every + /// source element count `src_meta`: + /// + /// ```text + /// dst_meta = dst_base + src_meta * (src.elem_size / dst.elem_size) + /// ``` + /// + /// This recognizes sufficient conditions for equality. A + /// `false` result also includes equivalent size sequences + /// that this test cannot recognize, or an adjustment whose + /// normalized components cannot be represented in `usize`. + const fn size_sequences_match( + src: TrailingSliceLayout, + dst: TrailingSliceLayout, + dst_base: usize, + ) -> bool { + let base_bytes = match dst_base.checked_mul(dst.elem_size) { + Some(bytes) => bytes, + None => return false, + }; + let shifted_dst = match dst.advance(base_bytes, src.elem_size) { + Some(layout) => layout, + None => return false, + }; + src.has_same_size_sequence(shifted_dst) + } + + /// Given the complete layouts of `Src` and `Dst`, produces + /// parameters for a cast that preserves object size. + /// + /// A `Some` result establishes that `Src`'s alignment is at + /// least `Dst`'s and that the metadata map preserves size for + /// every valid source metadata value. + /// + /// Supports casts between sized types of equal size, from a + /// sized type to a slice DST, and between slice DSTs. A slice + /// DST destination must have nonzero-sized elements; between + /// slice DSTs, its element size must exactly divide the + /// source's nonzero element size. + /// + /// Produces `None` for unsupported casts or when this method + /// cannot establish a size-preserving metadata map. Rejection + /// does not establish that no such map exists. const fn try_compute( - src: &DstLayout, - dst: &DstLayout, + src_layout: &DstLayout, + dst_layout: &DstLayout, ) -> Option> { - if src.align.get() < dst.align.get() { + if src_layout.align.get() < dst_layout.align.get() { return None; } - let inner = match (src.size_info, dst.size_info) { + let inner = match (src_layout.size_info, dst_layout.size_info) { ( SizeInfo::Sized { size: src_size }, SizeInfo::Sized { size: dst_size }, @@ -931,82 +2075,79 @@ mod cast_from { // dst_size`. CastParamsInner::SizedToSized } - (SizeInfo::Sized { size: src_size }, SizeInfo::SliceDst(dst)) => { - let offset_delta = if let Some(od) = src_size.checked_sub(dst.offset) { - od - } else { - return None; - }; - - let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) { - e - } else { - return None; + (SizeInfo::Sized { size: src_size }, SizeInfo::SliceDst(_)) => { + let dst_meta = match dst_layout.metadata_for_exact_size(src_size) { + Some(meta) => meta, + None => return None, }; - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let delta_mod_other_elem = offset_delta % dst_elem_size.get(); - - if delta_mod_other_elem != 0 { - return None; - } - - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let dst_meta = offset_delta / dst_elem_size.get(); - // SAFETY: The preceding math ensures that a `Dst` // with `dst_meta` addresses `src_size` bytes. CastParamsInner::SizedToUnsized { dst_meta } } (SizeInfo::SliceDst(src), SizeInfo::SliceDst(dst)) => { - let offset_delta = if let Some(od) = src.offset.checked_sub(dst.offset) - { - od - } else { - return None; - }; - let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) { e } else { return None; }; - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let delta_mod_other_elem = offset_delta % dst_elem_size.get(); - - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let elem_remainder = src.elem_size % dst_elem_size.get(); - - if delta_mod_other_elem != 0 - || src.elem_size < dst.elem_size - || elem_remainder != 0 - { + if src.elem_size < dst.elem_size { return None; } - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let offset_delta_elems = offset_delta / dst_elem_size.get(); + let (elem_multiple, described_src_elem_size) = + super::max_elems_for_bytes(src.elem_size, dst_elem_size); + if described_src_elem_size != src.elem_size { + return None; + } - // PANICS: `dst_elem_size: NonZeroUsize`, so this won't - // divide by zero. - #[allow(clippy::arithmetic_side_effects)] - let elem_multiple = src.elem_size / dst_elem_size.get(); + // Prefer an exact destination-element adjustment + // between `src.size_offset()` and + // `dst.size_offset()`. This preserves offset-based + // cast compatibility and metadata selection when + // several lengths have the same rounded size. It + // also avoids choosing a length whose subsequent + // padding sequence differs from the source's. + let exact_offset_delta_elems = + match src.size_offset().checked_sub(dst.size_offset()) { + Some(delta) => { + let (elems, described_delta) = + super::max_elems_for_bytes(delta, dst_elem_size); + if described_delta == delta { + Some(elems) + } else { + None + } + } + None => None, + }; + + let offset_delta_elems = match exact_offset_delta_elems { + Some(elems) if Self::size_sequences_match(src, dst, elems) => elems, + _ => { + let src_zero_size = match src.size_for_elems(0) { + Some(size) => size, + None => return None, + }; + let elems = + match dst_layout.metadata_for_exact_size(src_zero_size) { + Some(elems) => elems, + None => return None, + }; + if !Self::size_sequences_match(src, dst, elems) { + return None; + } + elems + } + }; CastParamsInner::UnsizedToUnsized { - // SAFETY: We checked above that this is an exact ratio. + // SAFETY: `size_sequences_match` proves that + // this affine metadata map preserves size. offset_delta_elems, - // SAFETY: We checked above that this is an exact ratio. + // SAFETY: We checked above that this is an exact + // ratio of source to destination element size. elem_multiple, } } @@ -1019,40 +2160,77 @@ mod cast_from { } impl CastParams { + /// Produces `Dst` metadata describing an object of the same + /// size as the `Src` described by `src_meta`. + /// /// # Safety /// /// `src_meta` describes a `Src` whose size is no larger than /// `isize::MAX`. - /// - /// The returned metadata describes a `Dst` of the same size as - /// the original `Src`. #[inline(always)] unsafe fn cast_metadata( self, src_meta: Src::PointerMetadata, ) -> Dst::PointerMetadata { - #[allow(unused)] - use crate::util::polyfills::*; - let dst_meta = match self.inner { CastParamsInner::UnsizedToUnsized { offset_delta_elems, elem_multiple } => { let src_meta = src_meta.to_elem_count(); - #[allow( - unstable_name_collisions, - clippy::multiple_unsafe_ops_per_block - )] - // SAFETY: `self` is a witness that the following - // equation holds: + // SAFETY: `self` witnesses that this affine map + // makes `Src` and `Dst`'s complete rounded size + // formulas equal for every source metadata value. + // Interpret the arithmetic in this proof over the + // mathematical nonnegative integers. Since the + // caller promises that `src_meta` is valid `Src` + // metadata, the source object size `src_size` it + // describes is at most `isize::MAX`. Let + // `src_elem_size` and `dst_elem_size` be the + // respective trailing-slice element sizes. Then: + // + // src_meta * src_elem_size + // <= src_size <= isize::MAX. + // + // Since `elem_multiple` is the exact ratio + // `src_elem_size / dst_elem_size` and + // `dst_elem_size >= 1`, // - // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM + // src_meta * elem_multiple + // <= src_meta * src_elem_size + // <= isize::MAX <= usize::MAX. // - // Since the caller promises that `src_meta` is - // valid `Src` metadata, this math will not - // overflow, and the returned value will describe a - // `Dst` of the same size. + // Thus `scaled_elems = src_meta * elem_multiple` + // is representable, and its byte contribution is + // `scaled_elems * dst_elem_size = src_meta * + // src_elem_size`. Let `base_bytes = + // offset_delta_elems * dst_elem_size`. + // `size_sequences_match` compares the source + // formula with the destination formula advanced by + // `base_bytes`. The latter's unrounded term + // contains `base_bytes + src_meta * src_elem_size`, + // and its complete size is at least that large + // (formally checked by Kani in + // `prove_size_formula_bounds_size_offset`). + // Sequence equality therefore gives, over the + // mathematical nonnegative integers, + // + // base_bytes + src_meta * src_elem_size + // = (offset_delta_elems + scaled_elems) + // * dst_elem_size + // <= src_size. + // + // Since `dst_elem_size >= 1`, the metadata sum + // `offset_delta_elems + scaled_elems` is also at + // most `src_size`, and thus at most `isize::MAX <= + // usize::MAX`. The sum is therefore representable, + // and the returned metadata describes a `Dst` of + // the same size. + // SAFETY: The bounds above establish the helper's + // complete no-overflow precondition. unsafe { - offset_delta_elems - .unchecked_add(src_meta.unchecked_mul(elem_multiple)) + super::add_scaled_metadata( + offset_delta_elems, + src_meta, + elem_multiple, + ) } } CastParamsInner::SizedToUnsized { dst_meta } => dst_meta, @@ -1095,6 +2273,55 @@ mod cast_from { } } +#[cfg(any(test, all(kani, feature = "derive")))] +mod padding_testutil { + use crate::KnownLayout; + + #[derive(KnownLayout)] + #[repr(C, align(4))] + pub(super) struct Aligned { + pub(super) prefix: Prefix, + pub(super) tail: Tail, + } + + /// Compare direct padding with the complete size minus the slice end. + /// Counts whose complete object size exceeds `usize::MAX` are omitted. + pub(super) fn check_padding_for_elems(elems: usize) -> Option + where + Target: ?Sized + KnownLayout, + { + let layout = crate::trailing_slice_layout::(); + let trailing_size = elems.checked_mul(layout.elem_size)?; + + // Evaluate the complete rounded size independently of both helpers: + // `size_for_elems` delegates to the padding method being tested. + let (alignment, phase) = layout.size_rounding_align_and_phase.components(); + #[allow(clippy::arithmetic_side_effects)] + let mask = alignment.get() - 1; + let unrounded = phase.checked_add(trailing_size)?; + let rounded = unrounded.checked_add(mask)? & !mask; + let object_size = layout.size_base.checked_add(rounded)?; + let trailing_end = layout.offset.checked_add(trailing_size).unwrap(); + let expected = object_size.checked_sub(trailing_end).unwrap(); + + let padding = layout.padding_for_elems(elems); + assert_eq!(padding, expected); + Some(padding) + } + + pub(super) fn check_layouts(elems: usize) { + // Slices have no trailing padding. The aligned fixtures cover + // zero and nonzero phases, a three-byte element whose padding + // cycles with the element count, and zero-sized elements. + let _ = check_padding_for_elems::<[u8]>(elems); + let _ = check_padding_for_elems::<[()]>(elems); + let _ = check_padding_for_elems::>(elems); + let _ = check_padding_for_elems::>(elems); + let _ = check_padding_for_elems::>(elems); + let _ = check_padding_for_elems::>(elems); + } +} + // FIXME(#67): For some reason, on our MSRV toolchain, this `allow` isn't // enforced despite having `#![allow(unknown_lints)]` at the crate root, but // putting it here works. Once our MSRV is high enough that this bug has been @@ -1103,12 +2330,560 @@ mod cast_from { #[cfg(test)] mod tests { use super::*; + use crate::PointerMetadata as _; + + /// Models the size of nested `repr(C)` structs ending in a slice. + /// + /// `leading` lists structs from outermost to innermost. Each tuple gives + /// the packing factor, minimum alignment, and prefix size before padding + /// for the final field. `trailing_elem_size` and `trailing_alignment` give + /// the slice element size and alignment. An empty `leading` describes the + /// slice itself. + /// + /// The returned closure maps a slice length to the complete size, including + /// padding at every nesting level. It returns `None` only on `usize` + /// overflow; sizes above `isize::MAX` are permitted by this arithmetic model. + /// Packing affects field placement, preserving each field's internal + /// padding, even for descriptors that Rust would reject as types. + /// + /// # Panics + /// + /// Panics if any alignment or packing factor is not a power of two, a + /// minimum alignment exceeds its packing factor, or the element size is + /// not a multiple of the slice alignment. + fn size_for_metadata_model( + leading: &[(NonZeroUsize, NonZeroUsize, usize)], + trailing_elem_size: usize, + trailing_alignment: NonZeroUsize, + ) -> impl Fn(usize) -> Option + '_ { + // Rust requires power-of-two alignments and sizes divisible by their + // alignment [1]; alignment modifiers also take powers of two [2]. + // + // [1] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#size-and-alignment + // [2] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#the-alignment-modifiers + assert!(trailing_alignment.get().is_power_of_two()); + assert_eq!(trailing_elem_size % trailing_alignment, 0); + for &(packed, align, _) in leading { + assert!(packed.get().is_power_of_two()); + assert!(align.get().is_power_of_two()); + // The descriptor requires its minimum to fit under the packing + // cap. It also models combinations that Rust forbids, including + // simultaneous `align` and `packed` modifiers [2]. + assert!(align <= packed); + } + + move |elems| { + // A slice shares its array section's layout [3]. Array size is + // element size times length, with the element's alignment [4]. + // + // [3] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#slice-layout + // [4] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#array-layout + let mut size = trailing_elem_size.checked_mul(elems)?; + let mut alignment = trailing_alignment; + + // Work outward with each field's complete size, including its + // trailing padding: an enclosing representation "does not change + // the layout of the fields themselves" [5]. Thus packing a nested + // field preserves the padding already included in `size`. + // + // [5] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#representations + for &(packed, align, prefix) in leading.iter().rev() { + // Packing caps the field's alignment for placement [2]. + let field_align = alignment.min(packed); + + // `repr(C)` places the next field at an aligned offset [6]; + // packing requires the least sufficient inter-field padding + // [2]. Here `prefix` is the end of the preceding fields. + // + // [6] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#reprc-structs + let offset = prefix.checked_add(util::padding_needed_for(prefix, field_align))?; + + // `repr(C)` uses the greatest field alignment [6], and an + // explicit alignment can only raise it [2]. Here `align` + // summarizes the minimum from the prefix and any explicit + // alignment. Both operands are <= `packed`, so their maximum + // also respects the packing cap. + alignment = align.max(field_align); + + // Advance past the complete field, then round to the struct's + // alignment, following `repr(C)`'s final sizing steps [6]. + size = offset.checked_add(size)?; + size = size.checked_add(util::padding_needed_for(size, alignment))?; + } + // Each enclosing size is at least its field's size. Thus any + // checked multiplication or addition that overflowed above would + // also make the final size unrepresentable in `usize`. + Some(size) + } + } + + /// Computes [`size_for_metadata_model`]'s size function using `DstLayout`. + /// + /// Layout construction uses [`DstLayout::for_repr_c_struct`], and the + /// returned closure uses [`crate::PointerMetadata::size_for_metadata`]. + /// + /// # Panics + /// + /// Panics for invalid descriptors as in [`size_for_metadata_model`], or if + /// `DstLayout` cannot construct the layout. This includes static size + /// overflow and, in debug builds, alignments or packing factors above + /// [`DstLayout::CURRENT_MAX_ALIGN`]. + fn size_for_metadata_via_dst_layout( + leading: &[(NonZeroUsize, NonZeroUsize, usize)], + trailing_elem_size: usize, + trailing_alignment: NonZeroUsize, + ) -> impl Fn(usize) -> Option + '_ { + assert_eq!(trailing_elem_size % trailing_alignment, 0); + let mut layout = DstLayout { + align: trailing_alignment, + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 0, + elem_size: trailing_elem_size, + size_base: 0, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(trailing_alignment, 0), + }), + statically_shallow_unpadded: true, + }; + for &(packed, align, prefix) in leading.iter().rev() { + assert!(align <= packed); + let prefix = DstLayout { + align: DstLayout::MIN_ALIGN, + size_info: SizeInfo::Sized { size: prefix }, + statically_shallow_unpadded: true, + }; + layout = DstLayout::for_repr_c_struct(Some(align), Some(packed), &[prefix, layout]); + } + move |elems: usize| elems.size_for_metadata(layout) + } + + #[test] + fn test_model_via_dst_layout() { + let nz = |n| NonZeroUsize::new(n).unwrap(); + let cases: &[&[_]] = &[ + &[], + &[(nz(8), nz(1), 3)], + &[(nz(8), nz(8), 3)], + &[(nz(2), nz(1), 1), (nz(8), nz(8), 3)], + &[(nz(8), nz(8), 1), (nz(2), nz(1), 3), (nz(8), nz(8), 5)], + &[(nz(1), nz(1), 1), (nz(2), nz(2), 3), (nz(8), nz(8), 5)], + ]; + for &leading in cases { + for &(elem_size, alignment) in &[(0, 1), (0, 8), (1, 1), (2, 2), (3, 1), (8, 8)] { + let alignment = nz(alignment); + let expected = size_for_metadata_model(leading, elem_size, alignment); + let actual = size_for_metadata_via_dst_layout(leading, elem_size, alignment); + let max_elems = usize::MAX / elem_size.max(1); + for elems in (0..32).chain([ + max_elems - 1, + max_elems, + max_elems.saturating_add(1), + usize::MAX, + ]) { + assert_eq!( + actual(elems), + expected(elems), + "leading: {:?}, trailing: {:?}, elems: {}", + leading, + (elem_size, alignment), + elems, + ); + } + } + } + + for prefix in [usize::MAX - 1, usize::MAX] { + let leading = [(nz(1), nz(1), prefix)]; + let expected = size_for_metadata_model(&leading, 1, nz(1)); + let actual = size_for_metadata_via_dst_layout(&leading, 1, nz(1)); + for elems in [0, 1, 2] { + assert_eq!(actual(elems), expected(elems)); + } + } + } + + // This exercises arithmetic rather than unsafe operations; keep the + // smaller comparison test enabled under Miri instead of this large search. + #[test] + #[cfg_attr(miri, ignore)] + fn test_model_via_dst_layout_combinations() { + use rand::{rngs::SmallRng, Rng as _, SeedableRng as _}; + + fn check( + leading: &[(NonZeroUsize, NonZeroUsize, usize)], + trailing_elem_size: usize, + trailing_alignment: NonZeroUsize, + ) { + let expected = size_for_metadata_model(leading, trailing_elem_size, trailing_alignment); + assert!( + expected(0).is_some(), + "unrepresentable test layout: {:?}, {:?}", + leading, + (trailing_elem_size, trailing_alignment) + ); + let actual = std::panic::catch_unwind(|| { + size_for_metadata_via_dst_layout(leading, trailing_elem_size, trailing_alignment) + }) + .unwrap_or_else(|_| { + panic!( + "DstLayout rejected leading: {:?}, trailing: {:?}", + leading, + (trailing_elem_size, trailing_alignment) + ) + }); + let compare = |elems| { + let expected = expected(elems); + assert_eq!( + actual(elems), + expected, + "leading: {:?}, trailing: {:?}, elems: {}", + leading, + (trailing_elem_size, trailing_alignment), + elems, + ); + expected + }; + + let first_sixteen = 0..=16; + let pointer_width = (0..POINTER_WIDTH_BITS).step_by(4).flat_map(|shift| { + let elems = 1usize << shift; + [elems - 1, elems, elems + 1] + }); + let last_two = (usize::MAX - 1)..=usize::MAX; + let elemses = first_sixteen.chain(pointer_width).chain(last_two); + for elems in elemses { + let _ = compare(elems); + } + + // The modeled size is nondecreasing. Find the last fitting length, + // comparing every probe, then check both sides of that boundary. + // Zero-sized elements can remain representable at usize::MAX. + let (mut low, mut high) = (0, usize::MAX); + while low < high { + let distance = high - low; + let mid = low + distance / 2 + distance % 2; + if compare(mid).is_some() { + low = mid; + } else { + high = mid - 1; + } + } + assert!(compare(low).is_some()); + if let Some(previous) = low.checked_sub(1) { + assert!(compare(previous).is_some()); + } + if let Some(next) = low.checked_add(1) { + assert_eq!(compare(next), None); + } + } + + let nz = |n| NonZeroUsize::new(n).unwrap(); + let check_small_tails = |leading: &[_]| { + for alignment in [1, 2, 4, 8] { + for multiple in [0, 1, 2, 3, 7] { + check(leading, multiple * alignment, nz(alignment)); + } + } + }; + let mut fragments = Vec::new(); + for packed in [1, 2, 4] { + for align in [1, 2, 4].iter().copied().filter(|&align| align <= packed) { + for prefix in 0..=4 { + fragments.push((nz(packed), nz(align), prefix)); + } + } + } + + // Exhaust every combination in this small domain at depths 0, 1, 2. + check_small_tails(&[]); + for &outer in &fragments { + check_small_tails(&[outer]); + for &inner in &fragments { + check_small_tails(&[outer, inner]); + } + } + + let mut rng = SmallRng::seed_from_u64(0); + for depth in (0usize..=128).chain([256, 512, 1024]) { + // A small prefix adds at most 4 * max_align bytes per layer. + // Reserve half of usize::MAX for a large outermost prefix so + // construction remains representable even at the largest depth. + let align_limit = + DstLayout::CURRENT_MAX_ALIGN.get().min(usize::MAX / (8 * depth.max(1))); + let alignments: Vec<_> = (0..POINTER_WIDTH_BITS) + .map(|shift| 1usize << shift) + .take_while(|&align| align <= align_limit) + .collect(); + let last = alignments.len() - 1; + let max_align = alignments[last]; + for case in 0..40 { + let mut leading = Vec::with_capacity(depth); + for level in 0..depth { + // Mix arbitrary modifiers, increasing and decreasing + // alignments, alternating packing, and fully packed trees. + let (packed, align) = match case / 8 { + 0 => { + let packed = rng.gen_range(0..=last); + (packed, rng.gen_range(0..=packed)) + } + 1 => (last, level % alignments.len()), + 2 => (last, last - level % alignments.len()), + 3 => { + let index = if level % 2 == 0 { 0 } else { last }; + (index, index) + } + _ => (0, 0), + }; + let (packed, align) = (alignments[packed], alignments[align]); + let prefix = if level == 0 && case % 4 == 3 { + usize::MAX / 2 + rng.gen_range(0..=2) - 1 + } else { + match rng.gen_range(0..8) { + 0 => 0, + 1 => 1, + 2 => align - 1, + 3 => align, + 4 => align + 1, + 5 => packed - 1, + 6 => packed + 1, + _ => rng.gen_range(0..=2 * max_align), + } + }; + leading.push((nz(packed), nz(align), prefix)); + } + let alignment = alignments[rng.gen_range(0..=last)]; + let elem_size = match case % 8 { + 0 => 0, + 1 => alignment, + 2 => 2 * alignment, + 3 => 3 * alignment, + 4 => 7 * alignment, + 5 => 15 * alignment, + 6 => (usize::MAX / 2 / alignment) * alignment, + _ => (usize::MAX / alignment) * alignment, + }; + check(&leading, elem_size, nz(alignment)); + } + } + } + + const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_ALIGN: NonZeroUsize = match NonZeroUsize::new(8) { + Some(align) => align, + None => const_unreachable!(), + }; + const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE: RoundingAlignAndPhase = + RoundingAlignAndPhase::new(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_ALIGN, 3); + const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_ALIGN: usize = + TEST_SIZE_ROUNDING_ALIGN_AND_PHASE.align().get(); + const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_PHASE: usize = + TEST_SIZE_ROUNDING_ALIGN_AND_PHASE.components().1; + + #[test] + #[allow(clippy::arithmetic_side_effects)] + fn test_size_rounding_align_and_phase_encoding() { + // These constants ensure that encoding and decoding remain evaluable + // on every supported compiler, including the MSRV. + assert_eq!(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_ALIGN, 8); + assert_eq!(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_PHASE, 3); + + for shift in 0..POINTER_WIDTH_BITS { + #[allow(clippy::arithmetic_side_effects)] + let align = NonZeroUsize::new(1usize << shift).unwrap(); + for phase in [0, align.get() / 2, align.get() - 1] { + let rounding = RoundingAlignAndPhase::new(align, phase); + assert_eq!(rounding.align(), align); + assert_eq!(rounding.components().1, phase); + assert_eq!(rounding.0.get(), align.get() | phase); + } + } + + let max_align = DstLayout::THEORETICAL_MAX_ALIGN; + let max_phase = max_align.get() - 1; + let max = RoundingAlignAndPhase::new(max_align, max_phase); + assert_eq!(max.0.get(), usize::MAX); + assert_eq!(max.align(), max_align); + assert_eq!(max.components().1, max_phase); + + assert_eq!(mem::size_of::(), 4 * mem::size_of::()); + } + + #[test] + #[allow(clippy::arithmetic_side_effects)] + fn test_trailing_slice_layout_normal_form() { + // Compare the stored base-and-phase formula with + // `prefix + round_up(offset + elems * elem_size, align)`, where + // `prefix < align`. Move whole alignment-sized chunks of `offset` + // into `size_base` and retain the remainder as the phase. Also verify + // that `size_offset()` recovers the exact `offset` used by casts to + // choose between metadata values with the same rounded object size. + for align in [1, 2, 4, 8, 16] { + let align = NonZeroUsize::new(align).unwrap(); + for prefix in 0..align.get() { + for offset in 0..32 { + let phase = offset % align; + let size_base = (offset - phase) | prefix; + let layout = TrailingSliceLayout { + offset: 0, + elem_size: 3, + size_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, phase), + }; + assert_eq!(layout.size_offset(), offset); + + for elems in 0..32 { + let unrounded = offset + elems * layout.elem_size; + let old_size = + prefix + unrounded + util::padding_needed_for(unrounded, align); + assert_eq!(layout.size_for_elems(elems), Some(old_size)); + } + } + } + } + } + + #[test] + #[allow(clippy::arithmetic_side_effects)] + fn test_normalized_size_bounds() { + // Use checked evaluation as an independent oracle, including layouts + // whose empty object is already too large to represent. + let checked_size = |layout: TrailingSliceLayout, trailing_bytes: usize| { + let (alignment, phase) = layout.size_rounding_align_and_phase.components(); + let unrounded = phase.checked_add(trailing_bytes)?; + let rounded = unrounded.checked_add(util::padding_needed_for(unrounded, alignment))?; + layout.size_base.checked_add(rounded) + }; + for exponent in 0..POINTER_WIDTH_BITS { + let alignment = NonZeroUsize::new(1usize << exponent).unwrap(); + for phase in [0, alignment.get() / 2, alignment.get() - 1] { + for base in [0, 1, alignment.get() - 1, alignment.get(), usize::MAX - 1, usize::MAX] + { + let mut layout = TrailingSliceLayout { + offset: 0, + elem_size: 1, + size_base: base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(alignment, phase), + }; + for available in [0, 1, alignment.get() - 1, alignment.get(), usize::MAX] { + match layout.max_trailing_bytes(available) { + Some(max_bytes) => { + assert!(checked_size(layout, max_bytes).unwrap() <= available); + if let Some(next_bytes) = max_bytes.checked_add(1) { + assert!(checked_size(layout, next_bytes) + .map(|size| size > available) + .unwrap_or(true)); + } + } + None => { + assert!(checked_size(layout, 0) + .map(|size| size > available) + .unwrap_or(true)); + } + } + } + + for elem_size in [0, 1, 3, alignment.get(), usize::MAX] { + layout.elem_size = elem_size; + let max_elems = layout + .max_trailing_bytes(usize::MAX) + .and_then(|bytes| bytes.checked_div(elem_size)) + .unwrap_or(usize::MAX); + for elems in [ + 0, + 1, + max_elems.saturating_sub(1), + max_elems, + max_elems.saturating_add(1), + usize::MAX, + ] { + let expected = elems + .checked_mul(elem_size) + .and_then(|bytes| checked_size(layout, bytes)); + // The physical slice offset does not affect + // the size formula. Raw layouts may place the + // slice beyond the object, and adding its byte + // count to the offset may overflow. + for offset in [0, usize::MAX] { + layout.offset = offset; + assert_eq!(layout.size_for_elems(elems), expected); + } + } + } + } + } + } + } + + #[test] + fn test_padding_for_elems_for_rust_layouts() { + use super::padding_testutil::{check_layouts, check_padding_for_elems, Aligned}; + + for elems in 0..256 { + check_layouts(elems); + } + let size_limit = usize::MAX; + for remaining in 0..16 { + check_layouts(size_limit - remaining); + check_layouts(size_limit / 3 - remaining); + } + check_layouts(usize::MAX); + + for (elems, expected) in [(0, 3), (1, 0), (2, 1), (3, 2)] { + assert_eq!( + check_padding_for_elems::>(elems), + Some(expected) + ); + } + // Every count is valid for zero-sized trailing elements. + assert_eq!(check_padding_for_elems::>(usize::MAX), Some(3)); + } + + #[test] + fn test_padding_for_elems() { + for exponent in 0..POINTER_WIDTH_BITS { + let alignment = NonZeroUsize::new(1usize << exponent).unwrap(); + let mask = alignment.get() - 1; + for phase in [0, alignment.get() / 2, mask] { + // Include zero-sized elements, a negative fixed term, fixed + // padding larger than the alignment, and overflowing sums + // and products. Raw layouts need not describe Rust types. + for (offset, elem_size, size_base) in [ + (0, 0, 0), + (0, 1, 1), + (9, 3, 8), + (3, 3, 0), + (9, 3, 12), + (usize::MAX, usize::MAX, usize::MAX), + (usize::MAX, 0, usize::MAX), + ] { + let layout = TrailingSliceLayout { + offset, + elem_size, + size_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(alignment, phase), + }; + for elems in [0, 1, 2, 3, alignment.get(), usize::MAX] { + // Independently reconstruct the complete rounded + // size and slice end, then subtract. Round up by + // adding the mask and clearing the low bits. + let trailing_bytes = elems.wrapping_mul(elem_size); + let unrounded = phase.wrapping_add(trailing_bytes); + let rounded = unrounded.wrapping_add(mask) & !mask; + let object_size = size_base.wrapping_add(rounded); + let trailing_end = offset.wrapping_add(trailing_bytes); + assert_eq!( + layout.padding_for_elems(elems), + object_size.wrapping_sub(trailing_end), + ); + } + } + } + } + } #[test] fn test_dst_layout_for_slice() { let layout = DstLayout::for_slice::(); match layout.size_info { - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => { + SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size, .. }) => { assert_eq!(offset, 0); assert_eq!(elem_size, 4); } @@ -1121,17 +2896,18 @@ mod tests { #[allow(clippy::decimal_literal_representation)] #[test] fn test_dst_layout_extend_sized_with_sized() { - // This macro constructs a layout corresponding to a `u8` and extends it - // with a zero-sized trailing field of given alignment `n`. The macro - // tests that the resulting layout has both size and alignment `min(n, - // P)` for all valid values of `repr(packed(P))`. + // This macro constructs a layout corresponding to a `u8` and extends + // it with a zero-sized trailing field of the specified alignment. The + // resulting size and alignment must both be the field's alignment + // capped by the packing limit. Exercise every supported packing limit + // as well as the absence of a packing limit. macro_rules! test_align_is_size { ($n:expr) => { let base = DstLayout::for_type::(); let trailing_field = DstLayout::for_type::>(); - let packs = - core::iter::once(None).chain((0..29).map(|p| NonZeroUsize::new(2usize.pow(p)))); + let packs = core::iter::once(None) + .chain((0..=29).map(|p| NonZeroUsize::new(2usize.pow(p)))); for pack in packs { let composite = base.extend(trailing_field, pack); @@ -1179,6 +2955,7 @@ mod tests { test_align_is_size!(33554432); test_align_is_size!(134217728); test_align_is_size!(268435456); + test_align_is_size!(536870912); } /// Tests of when a sized `DstLayout` is extended with a DST field. @@ -1192,15 +2969,29 @@ mod tests { let aligns = (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap()); let packs = core::iter::once(None).chain(aligns.clone().map(Some)); - for align in aligns { + for field_type_align in aligns { for pack in packs.clone() { let base = DstLayout::for_type::(); let elem_size = 42; let trailing_field_offset = 11; + #[allow(clippy::arithmetic_side_effects)] + let trailing_phase = trailing_field_offset % field_type_align; + let trailing_base = util::round_down_to_next_multiple_of_alignment( + trailing_field_offset, + field_type_align, + ); let trailing_field = DstLayout { - align, - size_info: SizeInfo::SliceDst(TrailingSliceLayout { elem_size, offset: 11 }), + align: field_type_align, + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + elem_size, + offset: trailing_field_offset, + size_base: trailing_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + field_type_align, + trailing_phase, + ), + }), statically_shallow_unpadded: false, }; @@ -1208,7 +2999,9 @@ mod tests { let max_align = pack.unwrap_or(DstLayout::CURRENT_MAX_ALIGN).get(); - let align = align.get().min(max_align); + let align = field_type_align.get().min(max_align); + let field_offset = align; + let size_base = field_offset + trailing_base; assert_eq!( composite, @@ -1216,7 +3009,12 @@ mod tests { align: NonZeroUsize::new(align).unwrap(), size_info: SizeInfo::SliceDst(TrailingSliceLayout { elem_size, - offset: align + trailing_field_offset, + offset: field_offset + trailing_field_offset, + size_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + field_type_align, + trailing_phase, + ), }), statically_shallow_unpadded: false, } @@ -1292,7 +3090,8 @@ mod tests { => padded { size: current_max_align * 2, align: current_max_align }); } - /// Tests that calling `pad_to_align` on a DST `DstLayout` is a no-op. + /// Tests that calling `pad_to_align` on a DST `DstLayout` incorporates the + /// type's trailing-padding operation into its runtime size formula. #[test] fn test_dst_layout_pad_to_align_with_dst() { for align in (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap()) { @@ -1300,42 +3099,183 @@ mod tests { for elem_size in 0..10 { let layout = DstLayout { align, - size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base: offset, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + DstLayout::MIN_ALIGN, + 0, + ), + }), statically_shallow_unpadded: false, }; - assert_eq!(layout.pad_to_align(), layout); + assert_eq!( + layout.pad_to_align(), + DstLayout { + align, + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base: offset - offset % align, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + align, + offset % align + ), + }), + statically_shallow_unpadded: false, + } + ); } } } - } - - // This test takes a long time when running under Miri, so we skip it in - // that case. This is acceptable because this is a logic test that doesn't - // attempt to expose UB. - #[test] - #[cfg_attr(miri, ignore)] - fn test_validate_cast_and_convert_metadata() { - #[allow(non_local_definitions)] - impl From for SizeInfo { - fn from(size: usize) -> SizeInfo { - SizeInfo::Sized { size } - } - } - - #[allow(non_local_definitions)] - impl From<(usize, usize)> for SizeInfo { - fn from((offset, elem_size): (usize, usize)) -> SizeInfo { - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) - } - } - fn layout>(s: S, align: usize) -> DstLayout { + // Exercise the branch where the existing inner rounding alignment is + // at least the enclosing alignment. The inner rounded term is already + // aligned to two, so padding only advances `size_base` from three to + // four. + let inner_rounding_dominates = DstLayout { + align: NonZeroUsize::new(2).unwrap(), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 3, + elem_size: 1, + size_base: 3, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 1, + ), + }), + statically_shallow_unpadded: false, + }; + assert_eq!( + inner_rounding_dominates.pad_to_align(), DstLayout { - size_info: s.into(), - align: NonZeroUsize::new(align).unwrap(), + align: NonZeroUsize::new(2).unwrap(), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 3, + elem_size: 1, + size_base: 4, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 1 + ), + }), statically_shallow_unpadded: false, } - } + ); + + // Exercise the branch where the enclosing alignment is larger. For + // a trailing-slice byte count `trailing_size`, rounding + // `6 + round_up(1 + trailing_size, 4)` to eight gives the equivalent + // formula `8 + round_up(1 + trailing_size, 8)`. + let outer_rounding_dominates = DstLayout { + align: NonZeroUsize::new(8).unwrap(), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 7, + elem_size: 1, + size_base: 6, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 1, + ), + }), + statically_shallow_unpadded: false, + }; + assert_eq!( + outer_rounding_dominates.pad_to_align(), + DstLayout { + align: NonZeroUsize::new(8).unwrap(), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 7, + elem_size: 1, + size_base: 8, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(8).unwrap(), + 1 + ), + }), + statically_shallow_unpadded: false, + } + ); + } + + #[test] + fn test_trailing_slice_layout_size_sequence_equivalence() { + // These are the two normalized descriptions produced by a + // `repr(C, packed(2))` wrapper around slice DSTs whose element types + // both have size four but alignments four and two respectively. Their + // formulas differ structurally, but both simplify to `2 + 4 * elems`. + let align4 = TrailingSliceLayout { + offset: 2, + elem_size: 4, + size_base: 2, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 0, + ), + }; + let align2 = TrailingSliceLayout { + offset: 2, + elem_size: 4, + size_base: 2, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(2).unwrap(), + 0, + ), + }; + assert!(align4.has_same_size_sequence(align2)); + for elems in 0..8 { + assert_eq!(align4.size_for_elems(elems), Some(2 + 4 * elems)); + assert_eq!(align4.size_for_elems(elems), align2.size_for_elems(elems)); + } + + let dynamically_padded = TrailingSliceLayout { elem_size: 1, ..align4 }; + assert!(!dynamically_padded.has_same_size_sequence(align2)); + } + + // This test takes a long time when running under Miri, so we skip it in + // that case. This is acceptable because this is a logic test that doesn't + // attempt to expose UB. + #[test] + #[cfg_attr(miri, ignore)] + fn test_validate_cast_and_convert_metadata() { + #[allow(non_local_definitions)] + impl From for SizeInfo { + fn from(size: usize) -> SizeInfo { + SizeInfo::Sized { size } + } + } + + #[allow(non_local_definitions)] + impl From<(usize, usize)> for SizeInfo { + fn from((offset, elem_size): (usize, usize)) -> SizeInfo { + SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base: offset, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + DstLayout::MIN_ALIGN, + 0, + ), + }) + } + } + + fn layout>(s: S, align: usize) -> DstLayout { + let align = NonZeroUsize::new(align).unwrap(); + let size_info = match s.into() { + SizeInfo::SliceDst(mut trailing) => { + let phase = trailing.size_base % align; + trailing.size_base = + util::round_down_to_next_multiple_of_alignment(trailing.size_base, align); + trailing.size_rounding_align_and_phase = + RoundingAlignAndPhase::new(align, phase); + SizeInfo::SliceDst(trailing) + } + size_info => size_info, + }; + DstLayout { size_info, align, statically_shallow_unpadded: false } + } /// This macro accepts arguments in the form of: /// @@ -1531,10 +3471,18 @@ mod tests { let resulting_size = match layout.size_info { SizeInfo::Sized { size } => size, - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => { + SizeInfo::SliceDst(TrailingSliceLayout { + elem_size, + size_base, + size_rounding_align_and_phase, + .. + }) => { + let (size_align, size_phase) = size_rounding_align_and_phase.components(); let padded_size = |elems| { - let without_padding = offset + elems * elem_size; - without_padding + util::padding_needed_for(without_padding, align) + let without_padding = size_phase + elems * elem_size; + size_base + + without_padding + + util::padding_needed_for(without_padding, size_align) }; let resulting_size = padded_size(elems); @@ -1562,8 +3510,13 @@ mod tests { } else { let min_size = match layout.size_info { SizeInfo::Sized { size } => size, - SizeInfo::SliceDst(TrailingSliceLayout { offset, .. }) => { - offset + util::padding_needed_for(offset, layout.align) + SizeInfo::SliceDst(TrailingSliceLayout { + size_base, + size_rounding_align_and_phase, + .. + }) => { + let (size_align, size_phase) = size_rounding_align_and_phase.components(); + size_base + size_phase + util::padding_needed_for(size_phase, size_align) } }; @@ -1592,6 +3545,35 @@ mod tests { .map(|(size_info, align)| layout(size_info, align)); itertools::iproduct!(layouts, 0..8, 0..8, [CastType::Prefix, CastType::Suffix]) .for_each(validate_behavior); + + // Exercise a normalized formula which cannot be represented as + // `round_up(offset + elems * elem_size, align)`: the physical trailing + // slice begins at offset 7, while an inner aligned DST contributes the + // size formula `2 + round_up(5 + elems, 4)` to a packed outer type. + let nested_packed = DstLayout { + align: NonZeroUsize::new(2).unwrap(), + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset: 7, + elem_size: 1, + size_base: 6, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 1, + ), + }), + statically_shallow_unpadded: false, + }; + itertools::iproduct!([nested_packed], 0..8, 0..24, [CastType::Prefix, CastType::Suffix]) + .for_each(validate_behavior); + + assert!(matches!( + nested_packed.validate_cast_and_convert_metadata(0, 8, CastType::Prefix), + Err(MetadataCastError::Size) + )); + assert!(matches!( + nested_packed.validate_cast_and_convert_metadata(0, 10, CastType::Prefix), + Ok((3, 10)) + )); } #[test] @@ -1631,9 +3613,18 @@ mod tests { let dst = args.elem_size.is_some(); let layout = { let size_info = match args.elem_size { - Some(elem_size) => { - SizeInfo::SliceDst(TrailingSliceLayout { offset: args.offset, elem_size }) - } + Some(elem_size) => SizeInfo::SliceDst(TrailingSliceLayout { + offset: args.offset, + elem_size, + size_base: util::round_down_to_next_multiple_of_alignment( + args.offset, + args.align, + ), + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + args.align, + args.offset % args.align, + ), + }), None => SizeInfo::Sized { // Rust only supports types whose sizes are a multiple // of their alignment. If the macro created a type like @@ -1951,70 +3942,748 @@ mod tests { #[cfg(kani)] mod proofs { + //! Arithmetic models and input generators for the layout proofs. + //! + //! The contract models specify expected results without calling the + //! operation being checked. The composition models (`extended_layout`, + //! `padded_layout`, and `repr_c_layout`) also define the contracts' input + //! domains: `Some` supplies an expected layout, while `None` excludes an + //! input from the corresponding non-panicking contract. These models check + //! representability in `usize`, without imposing Rust's `isize::MAX` bound. + use core::alloc::Layout; use super::*; - impl kani::Arbitrary for DstLayout { - fn any() -> Self { - let align: NonZeroUsize = kani::any(); - let size_info: SizeInfo = kani::any(); - - kani::assume(align.is_power_of_two()); - kani::assume(align < DstLayout::THEORETICAL_MAX_ALIGN); - - // For testing purposes, we most care about instantiations of - // `DstLayout` that can correspond to actual Rust types. We use - // `Layout` to verify that our `DstLayout` satisfies the validity - // conditions of Rust layouts. - kani::assume( - match size_info { - SizeInfo::Sized { size } => Layout::from_size_align(size, align.get()), - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size: _ }) => { - // `SliceDst` cannot encode an exact size, but we know - // it is at least `offset` bytes. - Layout::from_size_align(offset, align.get()) + /// Evaluates the normalized object size for a trailing slice of + /// `trailing_size` bytes: + /// + /// ```text + /// size_base + round_up(size_phase + trailing_size, size_align) + /// ``` + /// + /// Here, `size_align` and `size_phase` come from `layout.size_rounding_align_and_phase`. + /// `round_up` rounds its input up to a multiple of its alignment. Returns + /// `None` if the size is not representable in `usize`. + /// + /// Using byte counts separates rounding from element-count multiplication + /// and permits proofs over byte counts that are not whole elements. + pub(super) fn slice_dst_size_for_trailing_bytes( + layout: TrailingSliceLayout, + trailing_size: usize, + ) -> Option { + let without_padding = + layout.size_rounding_align_and_phase.components().1.checked_add(trailing_size)?; + let padding = + util::padding_needed_for(without_padding, layout.size_rounding_align_and_phase.align()); + let rounded = without_padding.checked_add(padding)?; + layout.size_base.checked_add(rounded) + } + + /// Returns the least multiple of `align` greater than or equal to `bytes`, + /// or `None` if that multiple exceeds `usize::MAX`. + /// + /// Requires a power-of-two `align`. Composition models use this checked + /// operation to identify inputs for which adding padding would overflow. + fn round_up(bytes: usize, align: NonZeroUsize) -> Option { + bytes.checked_add(util::padding_needed_for(bytes, align)) + } + + /// Models the largest trailing-slice byte count whose object size fits in + /// `available` bytes. Returns `None` if even an empty trailing slice cannot + /// fit. The result need not be a multiple of the element size. + /// + /// Subtracting `size_base` leaves the budget for the rounded contribution. + /// Rounding that budget down to `size_align` and subtracting `size_phase` + /// gives the largest byte count whose `slice_dst_size_for_trailing_bytes` + /// result is at most `available`. Metadata and casting contracts use + /// this bound without calling the production capacity calculation. + pub(super) fn max_trailing_bytes( + trailing: TrailingSliceLayout, + available: usize, + ) -> Option { + let (align, phase) = trailing.size_rounding_align_and_phase.components(); + let capacity = available.checked_sub(trailing.size_base)?; + util::round_down_to_next_multiple_of_alignment(capacity, align).checked_sub(phase) + } + + /// Models appending `field` to a sized `prefix` for `DstLayout::extend`. + /// + /// Places the field at the first offset after the prefix that satisfies the + /// field's alignment, capped by `packed` when present. The resulting + /// alignment is the maximum of the prefix's alignment and this effective + /// field alignment. A DST field retains its inner rounding operation, with + /// its slice offset and size base shifted by the field's placement. + /// Trailing padding for the enclosing layout is left to `padded_layout`. + /// + /// Returns `None` for a non-power-of-two alignment, an unsized prefix, or + /// arithmetic overflow. The `statically_shallow_unpadded` flag is true only + /// if both inputs have it set and field placement introduces no gap. + pub(super) fn extended_layout( + prefix: DstLayout, + field: DstLayout, + packed: Option, + ) -> Option { + if !prefix.align.is_power_of_two() + || !field.align.is_power_of_two() + || packed.map_or(false, |align| !align.is_power_of_two()) + { + return None; + } + let SizeInfo::Sized { size: prefix_size } = prefix.size_info else { return None }; + let field_align = field.align.min(packed.unwrap_or(DstLayout::THEORETICAL_MAX_ALIGN)); + let field_offset = round_up(prefix_size, field_align)?; + let size_info = match field.size_info { + SizeInfo::Sized { size } => SizeInfo::Sized { size: field_offset.checked_add(size)? }, + SizeInfo::SliceDst(trailing) => SizeInfo::SliceDst(TrailingSliceLayout { + offset: field_offset.checked_add(trailing.offset)?, + size_base: field_offset.checked_add(trailing.size_base)?, + ..trailing + }), + }; + Some(DstLayout { + align: prefix.align.max(field_align), + size_info, + statically_shallow_unpadded: prefix.statically_shallow_unpadded + && field.statically_shallow_unpadded + && field_offset == prefix_size, + }) + } + + /// Models adding trailing padding for `DstLayout::pad_to_align`. + /// + /// A sized layout rounds its size up to a multiple of `layout.align`. A DST + /// instead composes that rounding with its existing size expression: + /// + /// ```text + /// round_up(size_base + round_up(size_phase + trailing_size, size_align), + /// layout.align) + /// ``` + /// + /// Here, `size_align` and `size_phase` come from the DST's + /// `size_rounding_align_and_phase`. The result encodes this expression with + /// one base, phase, and rounding alignment, preserving the physical slice + /// offset and element size. The `statically_shallow_unpadded` flag is + /// cleared only when a sized layout gains padding; DST padding is accounted + /// for by the size expression. + /// + /// Returns `None` if `layout.align` is not a power of two or a checked + /// intermediate overflows. The contract compares this expected encoding + /// with the production result; the separate padding formula proofs check + /// that the encoding preserves the rounded size. + pub(super) fn padded_layout(layout: DstLayout) -> Option { + if !layout.align.is_power_of_two() { + return None; + } + let (size_info, no_static_padding) = match layout.size_info { + SizeInfo::Sized { size } => { + let rounded = round_up(size, layout.align)?; + (SizeInfo::Sized { size: rounded }, rounded == size) + } + SizeInfo::SliceDst(trailing) => { + let (inner_align, phase) = trailing.size_rounding_align_and_phase.components(); + let (size_base, size_rounding_align_and_phase) = if inner_align >= layout.align { + ( + round_up(trailing.size_base, layout.align)?, + trailing.size_rounding_align_and_phase, + ) + } else { + let fixed = round_up(trailing.size_base, inner_align)?.checked_add(phase)?; + let new_phase = fixed & (layout.align.get() - 1); + (fixed - new_phase, RoundingAlignAndPhase::new(layout.align, new_phase)) + }; + ( + SizeInfo::SliceDst(TrailingSliceLayout { + size_base, + size_rounding_align_and_phase, + ..trailing + }), + true, + ) + } + }; + Some(DstLayout { + size_info, + statically_shallow_unpadded: layout.statically_shallow_unpadded && no_static_padding, + ..layout + }) + } + + /// Models a complete layout for `DstLayout::for_repr_c_struct` by composing + /// the field-placement and trailing-padding specifications. + /// + /// Starts with an empty prefix of alignment `align`, or one if absent. + /// Appends fields in order, with `packed` as an optional cap on field + /// alignment; the final layout is padded to its resulting alignment. Only + /// the last field may be unsized, since field extension requires a sized + /// prefix. + /// + /// Returns `None` if the initial alignment is not a power of two or any + /// field extension or final padding step returns `None`. + pub(super) fn repr_c_layout( + align: Option, + packed: Option, + fields: &[DstLayout], + ) -> Option { + let align = align.unwrap_or(DstLayout::MIN_ALIGN); + if !align.is_power_of_two() { + return None; + } + let mut layout = DstLayout { + align, + size_info: SizeInfo::Sized { size: 0 }, + statically_shallow_unpadded: true, + }; + for &field in fields { + layout = extended_layout(layout, field, packed)?; + } + padded_layout(layout) + } + + // The inline contract harnesses use unrestricted `Arbitrary` values. + // These generators retain the narrower domains of the Rust-layout proofs. + + /// Generates any power-of-two alignment representable in `usize`, including + /// `DstLayout::THEORETICAL_MAX_ALIGN`. This restricts the alignment alone; + /// callers must impose any constraints relating it to a layout's size. + fn any_layout_align() -> NonZeroUsize { + let exponent: u8 = kani::any(); + kani::assume(usize::from(exponent) < POINTER_WIDTH_BITS); + + // `exponent < POINTER_WIDTH_BITS`, so the shift is in range and its + // result is non-zero. This construction is surjective over every + // power-of-two alignment up to and including + // `THEORETICAL_MAX_ALIGN`. `Layout` admits that maximum alignment for + // zero-sized layouts, so the validity assumptions below—not this + // generator—filter combinations that cannot describe Rust types. + NonZeroUsize::new(1usize << exponent).unwrap() + } + + /// Generates layouts satisfying the arithmetic restrictions used by the + /// standalone Rust-layout proofs. + /// + /// The fixed size, or the zero-element DST size, must be accepted by + /// `Layout::from_size_align`. For a DST, `size_base + size_phase` must also + /// be representable and at least the physical slice offset. Component + /// bounds come from `any_bounded_size_info`; `statically_shallow_unpadded` + /// remains arbitrary. These conditions do not establish that a Rust type + /// with this layout exists. + fn any_valid_dst_layout() -> DstLayout { + let align = any_layout_align(); + let size_info = any_bounded_size_info(); + + kani::assume( + match size_info { + SizeInfo::Sized { size } => Layout::from_size_align(size, align.get()), + SizeInfo::SliceDst(trailing) => { + // `SliceDst` cannot encode one exact size. Validate its + // minimum size and the invariants required of the size + // formula for layouts of real Rust types. + let min_size = trailing.size_for_elems(0); + let unrounded_min_size = trailing + .size_base + .checked_add(trailing.size_rounding_align_and_phase.components().1); + + kani::assume(matches!( + unrounded_min_size, + Some(size) if size >= trailing.offset + )); + + match min_size { + Some(min_size) => Layout::from_size_align(min_size, align.get()), + None => Layout::from_size_align(usize::MAX, align.get()), } } - .is_ok(), - ); + } + .is_ok(), + ); + + DstLayout { align: align, size_info: size_info, statically_shallow_unpadded: kani::any() } + } + + /// Generates either a fixed size at most `DstLayout::MAX_SIZE`, or a DST + /// description from `any_bounded_trailing_layout`. + /// + /// Supplies bounded size components to `any_valid_dst_layout`, which adds + /// constraints involving the enclosing layout's alignment. + fn any_bounded_size_info() -> SizeInfo { + let is_sized: bool = kani::any(); + + match is_sized { + true => { + let size: usize = kani::any(); + + kani::assume(size <= DstLayout::MAX_SIZE); + + SizeInfo::Sized { size } + } + false => SizeInfo::SliceDst(any_bounded_trailing_layout()), + } + } + + /// Generates a trailing-slice description whose element size, slice offset, + /// and size base are each at most `DstLayout::MAX_SIZE` (`isize::MAX`). + /// + /// The rounding alignment spans all representable powers of two, and the + /// phase spans all values below that alignment. No further relationships + /// between these components are imposed: their combined size may overflow + /// or fail to contain the slice. Standalone proofs add the relationships + /// they require; contract harnesses use unrestricted `Arbitrary` values + /// instead. + fn any_bounded_trailing_layout() -> TrailingSliceLayout { + let elem_size: usize = kani::any(); + let offset: usize = kani::any(); + let size_base: usize = kani::any(); + let size_align = any_layout_align(); + let raw_size_phase: usize = kani::any(); + // Since `size_align` is a power of two, masking is surjective over + // precisely the values in `0..size_align`. + #[allow(clippy::arithmetic_side_effects)] + let size_phase = raw_size_phase & (size_align.get() - 1); + + kani::assume(elem_size <= DstLayout::MAX_SIZE); + kani::assume(offset <= DstLayout::MAX_SIZE); + kani::assume(size_base <= DstLayout::MAX_SIZE); + + TrailingSliceLayout { + elem_size, + offset, + size_base, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(size_align, size_phase), + } + } + + #[cfg(feature = "derive")] + #[kani::proof] + fn prove_padding_for_elems_for_rust_layouts() { + // For each fixture, check every element count whose complete size + // fits in `usize`, including all counts for zero-sized elements. + padding_testutil::check_layouts(kani::any()); + } + + #[cfg(kani_slow)] + #[kani::proof] + #[kani::solver(kissat)] + fn prove_padding_for_elems() { + // Check the modular result for every field value and element count, + // including layouts that cannot describe a Rust type. + let layout = TrailingSliceLayout { + offset: kani::any(), + elem_size: kani::any(), + size_base: kani::any(), + size_rounding_align_and_phase: RoundingAlignAndPhase(kani::any()), + }; + let elems: usize = kani::any(); + let (size_align, size_phase) = layout.size_rounding_align_and_phase.components(); + let mask = size_align.get() - 1; + + // Evaluate the complete size and trailing-slice end separately in + // modular arithmetic, without reducing the trailing bytes first or + // canceling them from the difference. + let trailing_bytes = elems.wrapping_mul(layout.elem_size); + let unrounded = size_phase.wrapping_add(trailing_bytes); + let rounded = unrounded.wrapping_add(mask) & !mask; + let object_size = layout.size_base.wrapping_add(rounded); + let trailing_end = layout.offset.wrapping_add(trailing_bytes); + assert_eq!(layout.padding_for_elems(elems), object_size.wrapping_sub(trailing_end)); + } + + #[kani::proof] + fn prove_size_rounding_align_and_phase_encoding() { + let encoded: usize = kani::any(); + kani::assume(encoded != 0); + let encoded = NonZeroUsize::new(encoded).unwrap(); + let rounding = RoundingAlignAndPhase(encoded); + + let align = rounding.align(); + let phase = rounding.components().1; + assert!(align.is_power_of_two()); + assert!(phase < align.get()); + assert_eq!(RoundingAlignAndPhase::new(align, phase), rounding); + } + + #[kani::proof] + fn prove_size_rounding_align_and_phase_decoding() { + let align: NonZeroUsize = kani::any(); + let phase: usize = kani::any(); + kani::assume(align.is_power_of_two()); + kani::assume(phase < align.get()); + + let rounding = RoundingAlignAndPhase::new(align, phase); + assert_eq!(rounding.components(), (align, phase)); + } + + #[cfg(kani_slow)] + #[kani::proof] + fn prove_trailing_slice_layout_normal_form() { + let rounding_word: usize = kani::any(); + kani::assume(rounding_word != 0); + let align = RoundingAlignAndPhase(NonZeroUsize::new(rounding_word).unwrap()).align(); + + let raw_prefix: usize = kani::any(); + let offset: usize = kani::any(); + let trailing_size: usize = kani::any(); + #[allow(clippy::arithmetic_side_effects)] + let prefix = raw_prefix & (align.get() - 1); + #[allow(clippy::arithmetic_side_effects)] + let phase = offset & (align.get() - 1); + #[allow(clippy::arithmetic_side_effects)] + let aligned_offset = offset - phase; + let layout = TrailingSliceLayout { + offset: 0, + elem_size: 1, + size_base: aligned_offset | prefix, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, phase), + }; + + assert_eq!(layout.size_offset(), offset); + + let old_size = offset + .checked_add(trailing_size) + .and_then(|without_padding| { + without_padding.checked_add(util::padding_needed_for(without_padding, align)) + }) + .and_then(|rounded| prefix.checked_add(rounded)); + assert_eq!(layout.size_for_elems(trailing_size), old_size); + assert_eq!(slice_dst_size_for_trailing_bytes(layout, trailing_size), old_size); + } + + #[cfg(kani_slow)] + #[kani::proof] + #[kani::solver(kissat)] + fn prove_element_product_preserves_alignment() { + let elem_size: usize = kani::any(); + let elems: usize = kani::any(); + let align = any_layout_align(); + #[allow(clippy::arithmetic_side_effects)] + let align_mask = align.get() - 1; + + kani::assume(elem_size & align_mask == 0); + let Some(bytes) = elem_size.checked_mul(elems) else { + kani::assume(false); + loop {} + }; + assert_eq!(bytes & align_mask, 0); + } + + #[kani::proof] + fn prove_trailing_slice_layout_advance() { + let layout: TrailingSliceLayout = any_bounded_trailing_layout(); + let bytes: usize = kani::any(); + let elem_size: usize = kani::any(); + let trailing_size: usize = kani::any(); + + let size_offset = layout.size_offset(); + let advanced_offset = size_offset.checked_add(bytes); + let advanced = layout.advance(bytes, elem_size); + assert_eq!(advanced.is_some(), advanced_offset.is_some()); + let Some(advanced_offset) = advanced_offset else { + kani::assume(false); + loop {} + }; + let Some(advanced) = advanced else { unreachable!() }; + + assert_eq!(advanced.offset, layout.offset); + assert_eq!(advanced.size_offset(), advanced_offset); + assert_eq!(advanced.elem_size, elem_size); + assert_eq!( + advanced.size_rounding_align_and_phase.align(), + layout.size_rounding_align_and_phase.align() + ); + #[allow(clippy::arithmetic_side_effects)] + let size_align_mask = advanced.size_rounding_align_and_phase.align().get() - 1; + assert_eq!(advanced.size_base & size_align_mask, layout.size_base & size_align_mask); + + let size_align = layout.size_rounding_align_and_phase.align(); + #[allow(clippy::arithmetic_side_effects)] + let size_prefix = layout.size_base & (size_align.get() - 1); + let expected_size = advanced_offset + .checked_add(trailing_size) + .and_then(|without_padding| { + without_padding.checked_add(util::padding_needed_for(without_padding, size_align)) + }) + .and_then(|rounded| size_prefix.checked_add(rounded)); + assert_eq!(slice_dst_size_for_trailing_bytes(advanced, trailing_size), expected_size); + } + + #[cfg(kani_slow)] + #[kani::proof] + fn prove_has_same_size_sequence() { + let left: TrailingSliceLayout = any_bounded_trailing_layout(); + let right: TrailingSliceLayout = any_bounded_trailing_layout(); + let trailing_size: usize = kani::any(); + + kani::assume(left.has_same_size_sequence(right)); + let left_align = left.size_rounding_align_and_phase.align().get(); + let right_align = right.size_rounding_align_and_phase.align().get(); + let max_align = if left_align > right_align { left_align } else { right_align }; + #[allow(clippy::arithmetic_side_effects)] + let max_align_mask = max_align - 1; + if left.elem_size & max_align_mask == 0 { + // In this branch `has_same_size_sequence` relies on every actual + // trailing byte count being a multiple of both alignments. Prove + // the formula for that entire byte class, which includes every + // representable `elems * elem_size`, without introducing a + // nonlinear symbolic multiplication into the harness. + kani::assume(trailing_size & max_align_mask == 0); + } + assert_eq!( + slice_dst_size_for_trailing_bytes(left, trailing_size), + slice_dst_size_for_trailing_bytes(right, trailing_size) + ); + } + + #[kani::proof] + #[kani::solver(kissat)] + fn prove_max_trailing_bytes() { + let trailing: TrailingSliceLayout = any_bounded_trailing_layout(); + let available_bytes: usize = kani::any(); + let trailing_bytes: usize = kani::any(); + let fits = slice_dst_size_for_trailing_bytes(trailing, trailing_bytes) + .map(|size| size <= available_bytes) + .unwrap_or(false); + match trailing.max_trailing_bytes(available_bytes) { + Some(max_bytes) => assert_eq!(trailing_bytes <= max_bytes, fits), + None => assert!(!fits), + } + } + + #[kani::proof] + #[kani::solver(kissat)] + fn prove_max_trailing_bytes_components() { + let trailing: TrailingSliceLayout = any_bounded_trailing_layout(); + let bytes_len: usize = kani::any(); + let Some(max_slice_bytes) = trailing.max_trailing_bytes(bytes_len) else { + kani::assume(false); + loop {} + }; + let (size_align, size_phase) = trailing.size_rounding_align_and_phase.components(); + let Some(bytes_after_base) = bytes_len.checked_sub(trailing.size_base) else { + unreachable!() + }; + let max_rounded_bytes = + util::round_down_to_next_multiple_of_alignment(bytes_after_base, size_align); + assert_eq!(size_phase.checked_add(max_slice_bytes), Some(max_rounded_bytes)); + assert!(trailing.size_base.checked_add(max_rounded_bytes).unwrap() <= bytes_len); + } + + #[kani::proof] + #[kani::solver(kissat)] + fn prove_slice_dst_validator_selected_size_fits() { + let size_align = any_layout_align(); + #[allow(clippy::arithmetic_side_effects)] + let align_mask = size_align.get() - 1; + let raw_phase: usize = kani::any(); + let size_phase = raw_phase & align_mask; + let raw_capacity: usize = kani::any(); + let max_rounded_bytes = raw_capacity & !align_mask; + let size_base: usize = kani::any(); + let bytes_len: usize = kani::any(); + let trailing_size: usize = kani::any(); + + // These are exactly the relations established for every successful + // capacity query by `prove_max_trailing_bytes_components`. Isolate + // them from the helper's implementation to keep this arithmetic + // identity independent of how that helper computes its bound. + let Some(max_slice_bytes) = max_rounded_bytes.checked_sub(size_phase) else { + kani::assume(false); + loop {} + }; + let Some(max_object_size) = size_base.checked_add(max_rounded_bytes) else { + kani::assume(false); + loop {} + }; + kani::assume(max_object_size <= bytes_len); + kani::assume(trailing_size <= max_slice_bytes); + + // For a whole-element count selected by the validator, this unused + // byte count is also `max_slice_bytes % elem_size`. The identity here + // holds for every byte count satisfying the capacity bound. + #[allow(clippy::arithmetic_side_effects)] + let unused_bytes = max_slice_bytes - trailing_size; + let unused_aligned_bytes = + util::round_down_to_next_multiple_of_alignment(unused_bytes, size_align); + #[allow(clippy::arithmetic_side_effects)] + let object_size = size_base + (max_rounded_bytes - unused_aligned_bytes); + + let unrounded = size_phase.checked_add(trailing_size).unwrap(); + let rounded = + unrounded.checked_add(util::padding_needed_for(unrounded, size_align)).unwrap(); + assert_eq!(Some(object_size), size_base.checked_add(rounded)); + assert!(object_size <= bytes_len); + } - Self { align: align, size_info: size_info, statically_shallow_unpadded: kani::any() } + #[kani::proof] + #[kani::solver(kissat)] + fn prove_slice_dst_validator_rejects_larger_trailing_size() { + let trailing: TrailingSliceLayout = any_bounded_trailing_layout(); + let bytes_len: usize = kani::any(); + let larger_trailing_size: usize = kani::any(); + let Some(max_slice_bytes) = trailing.max_trailing_bytes(bytes_len) else { + kani::assume(false); + loop {} + }; + kani::assume(larger_trailing_size > max_slice_bytes); + if let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, larger_trailing_size) + { + assert!(object_size > bytes_len); } } - impl kani::Arbitrary for SizeInfo { - fn any() -> Self { - let is_sized: bool = kani::any(); + #[kani::proof] + fn prove_slice_formula_contains_trailing_slice() { + let trailing: TrailingSliceLayout = any_bounded_trailing_layout(); + let trailing_size: usize = kani::any(); + let (_, size_phase) = trailing.size_rounding_align_and_phase.components(); + + let Some(unrounded_min) = trailing.size_base.checked_add(size_phase) else { + kani::assume(false); + loop {} + }; + kani::assume(unrounded_min >= trailing.offset); + let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else { + kani::assume(false); + loop {} + }; + + let Some(trailing_end) = trailing.offset.checked_add(trailing_size) else { unreachable!() }; + assert!(trailing_end <= object_size); + } + + #[kani::proof] + fn prove_finalized_slice_formula_is_aligned() { + let trailing: TrailingSliceLayout = any_bounded_trailing_layout(); + let outer_align = any_layout_align(); + let trailing_size: usize = kani::any(); + kani::assume(trailing.size_rounding_align_and_phase.align() >= outer_align); + #[allow(clippy::arithmetic_side_effects)] + let outer_align_mask = outer_align.get() - 1; + kani::assume(trailing.size_base & outer_align_mask == 0); + let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else { + kani::assume(false); + loop {} + }; - match is_sized { - true => { - let size: usize = kani::any(); + assert_eq!(object_size & outer_align_mask, 0); + } - kani::assume(size <= DstLayout::MAX_SIZE); + #[kani::proof] + fn prove_validator_checks_selected_endpoint_alignment() { + let align = any_layout_align(); + let addr: usize = kani::any(); + let bytes_len: usize = kani::any(); + let Some(end) = addr.checked_add(bytes_len) else { + kani::assume(false); + loop {} + }; + let suffix: bool = kani::any(); + let cast_type = if suffix { CastType::Suffix } else { CastType::Prefix }; + let selected_endpoint = if suffix { end } else { addr }; + #[allow(clippy::arithmetic_side_effects)] + let align_mask = align.get() - 1; - SizeInfo::Sized { size } + // The endpoint check precedes and is independent of the size-info + // branch. A zero-sized layout makes every aligned case succeed and + // isolates that control flow, including prefix/suffix split selection. + let layout = DstLayout { + align, + size_info: SizeInfo::Sized { size: 0 }, + statically_shallow_unpadded: false, + }; + match layout.validate_cast_and_convert_metadata(addr, bytes_len, cast_type) { + Ok((elems, split_at)) => { + assert_eq!(selected_endpoint & align_mask, 0); + assert_eq!(elems, 0); + if suffix { + assert_eq!(split_at, bytes_len); + } else { + assert_eq!(split_at, 0); } - false => SizeInfo::SliceDst(kani::any()), } + Err(MetadataCastError::Alignment) => { + assert_ne!(selected_endpoint & align_mask, 0); + } + Err(MetadataCastError::Size) => unreachable!(), } } - impl kani::Arbitrary for TrailingSliceLayout { - fn any() -> Self { - let elem_size: usize = kani::any(); - let offset: usize = kani::any(); + #[kani::proof] + fn prove_aligned_suffix_end_and_size_imply_aligned_start() { + let align = any_layout_align(); + #[allow(clippy::arithmetic_side_effects)] + let align_mask = align.get() - 1; + let addr: usize = kani::any(); + let bytes_len: usize = kani::any(); + let object_size: usize = kani::any(); + let Some(end) = addr.checked_add(bytes_len) else { + kani::assume(false); + loop {} + }; + kani::assume(end & align_mask == 0); + kani::assume(object_size & align_mask == 0); + kani::assume(object_size <= bytes_len); - kani::assume(elem_size < DstLayout::MAX_SIZE); - kani::assume(offset < DstLayout::MAX_SIZE); + #[allow(clippy::arithmetic_side_effects)] + let split_at = bytes_len - object_size; + let Some(object_start) = addr.checked_add(split_at) else { unreachable!() }; + assert_eq!(object_start & align_mask, 0); + } - TrailingSliceLayout { elem_size, offset } - } + #[cfg(kani_slow)] + #[kani::proof] + #[kani::solver(kissat)] + fn prove_size_formula_bounds_size_offset() { + // Prove the bound for arbitrary byte counts, including counts that + // are not whole elements. The `size_for_elems` contract relates this + // formula to checked element products. This harness checks the + // formula's bound independently of that implementation. + let trailing: TrailingSliceLayout = kani::any(); + let trailing_size: usize = kani::any(); + let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else { + kani::assume(false); + loop {} + }; + let size_offset = trailing.size_offset(); + let Some(without_padding) = size_offset.checked_add(trailing_size) else { unreachable!() }; + + assert!(without_padding <= object_size); + } + + // Older supported compilers resolve the unchecked operation syntax in + // `add_scaled_metadata` to `polyfills::NumExt`; newer compilers prefer the + // inherent methods. Use UFCS to verify the fallback implementation which + // Kani's modern rustc cannot select in the production helper. + #[kani::proof] + #[kani::solver(kissat)] + fn prove_num_ext_affine_fallback_matches_checked_arithmetic() { + let base: usize = kani::any(); + let metadata: usize = kani::any(); + let multiple: usize = kani::any(); + let Some(scaled) = metadata.checked_mul(multiple) else { + kani::assume(false); + loop {} + }; + let Some(expected) = base.checked_add(scaled) else { + kani::assume(false); + loop {} + }; + + // SAFETY: The checked multiplication returned `Some`, satisfying + // `NumExt::unchecked_mul`'s no-overflow contract. + let scaled = + unsafe { ::unchecked_mul(metadata, multiple) }; + // SAFETY: The checked addition returned `Some`, and `scaled` is the + // checked product, satisfying `NumExt::unchecked_add`'s no-overflow + // contract. + let actual = + unsafe { ::unchecked_add(base, scaled) }; + assert_eq!(actual, expected); } + #[cfg(kani_slow)] #[kani::proof] fn prove_requires_dynamic_padding() { - let layout: DstLayout = kani::any(); + let layout: DstLayout = any_valid_dst_layout(); let SizeInfo::SliceDst(size_info) = layout.size_info else { kani::assume(false); @@ -2029,31 +4698,35 @@ mod proofs { loop {} }; - let Some(unpadded_size) = size_info.offset.checked_add(trailing_slice_size) else { - // The `unpadded_size` exceeds `usize::MAX`; `meta`` is invalid. + let Some(trailing_slice_end) = size_info.offset.checked_add(trailing_slice_size) else { + // The trailing slice end exceeds `usize::MAX`; `meta` is invalid. kani::assume(false); loop {} }; - if unpadded_size >= DstLayout::MAX_SIZE { - // The `unpadded_size` exceeds `isize::MAX`; `meta` is invalid. + let Some(size) = slice_dst_size_for_trailing_bytes(size_info, trailing_slice_size) else { kani::assume(false); loop {} - } + }; - let trailing_padding = util::padding_needed_for(unpadded_size, layout.align); + if size > DstLayout::MAX_SIZE || trailing_slice_end > size { + // This metadata cannot describe a valid instance. + kani::assume(false); + loop {} + } if !layout.requires_dynamic_padding() { - assert!(trailing_padding == 0); + assert_eq!(trailing_slice_end, size); } } + #[cfg(kani_slow)] #[kani::proof] fn prove_dst_layout_extend() { use crate::util::{max, min, padding_needed_for}; - let base: DstLayout = kani::any(); - let field: DstLayout = kani::any(); + let base: DstLayout = any_valid_dst_layout(); + let field: DstLayout = any_valid_dst_layout(); let packed: Option = kani::any(); if let Some(max_align) = packed { @@ -2069,9 +4742,6 @@ mod proofs { unreachable!(); }; - // Under the above conditions, `DstLayout::extend` will not panic. - let composite = base.extend(field, packed); - // The field's alignment is clamped by `max_align` (i.e., the // `packed` attribute, if any) [1]. // @@ -2082,10 +4752,6 @@ mod proofs { // alignment of the field's type. let field_align = min(field.align, packed.unwrap_or(DstLayout::THEORETICAL_MAX_ALIGN)); - // The struct's alignment is the maximum of its previous alignment and - // `field_align`. - assert_eq!(composite.align, max(base.align, field_align)); - // Compute the minimum amount of inter-field padding needed to // satisfy the field's alignment, and offset of the trailing field. // [1] @@ -2095,7 +4761,38 @@ mod proofs { // Inter-field padding is guaranteed to be the minimum required in // order to satisfy each field's (possibly altered) alignment. let padding = padding_needed_for(base_size, field_align); - let offset = base_size + padding; + let offset = match base_size.checked_add(padding) { + Some(offset) => offset, + None => { + kani::assume(false); + loop {} + } + }; + + // Restrict the proof to extensions whose zero-metadata composite size + // is representable. This semantic condition implies the + // representability of each implementation intermediate below; prove + // those implications rather than assuming the intermediates. + match field.size_info { + SizeInfo::Sized { size } => kani::assume(offset.checked_add(size).is_some()), + SizeInfo::SliceDst(trailing) => { + let Some(field_min) = trailing.size_for_elems(0) else { + kani::assume(false); + loop {} + }; + kani::assume(offset.checked_add(field_min).is_some()); + + assert!(offset.checked_add(trailing.offset).is_some()); + assert!(offset.checked_add(trailing.size_base).is_some()); + } + } + + // Under the above conditions, `DstLayout::extend` will not panic. + let composite = base.extend(field, packed); + + // The struct's alignment is the maximum of its previous alignment and + // `field_align`. + assert_eq!(composite.align, max(base.align, field_align)); // For testing purposes, we'll also construct `alloc::Layout` // stand-ins for `DstLayout`, and show that `extend` behaves @@ -2132,29 +4829,64 @@ mod proofs { panic!("The composite of two sized layouts must be sized.") } } - SizeInfo::SliceDst(TrailingSliceLayout { - offset: field_offset, - elem_size: field_elem_size, - }) => { - if let SizeInfo::SliceDst(TrailingSliceLayout { - offset: composite_offset, - elem_size: composite_elem_size, - }) = composite.size_info - { + SizeInfo::SliceDst(field_trailing) => { + if let SizeInfo::SliceDst(composite_trailing) = composite.size_info { // The offset of the trailing slice component is the sum // of the offset of the trailing field and the trailing // slice offset within that field. - assert_eq!(composite_offset, offset + field_offset); + assert_eq!(composite_trailing.offset, offset + field_trailing.offset); // The elem size is unchanged. - assert_eq!(composite_elem_size, field_elem_size); + assert_eq!(composite_trailing.elem_size, field_trailing.elem_size); + // Extension preserves the invariant that the physical + // trailing-slice field fits within the unrounded minimum + // size. This is one of the premises consumed by the + // validator soundness proof. + let composite_phase = + composite_trailing.size_rounding_align_and_phase.components().1; + let Some(composite_unrounded_min) = + composite_trailing.size_base.checked_add(composite_phase) + else { + unreachable!() + }; + assert!(composite_unrounded_min >= composite_trailing.offset); + + // The normalized formula must continue to describe the + // size of the field at every trailing byte contribution + // for which the checked size computation succeeds. This + // is stronger than considering only products of a slice + // length and `elem_size`, and avoids a nonlinear + // multiplication in the verifier. + let trailing_size: usize = kani::any(); + let field_size = + match slice_dst_size_for_trailing_bytes(field_trailing, trailing_size) { + Some(size) => size, + None => { + kani::assume(false); + loop {} + } + }; + let expected_size = match offset.checked_add(field_size) { + Some(size) => size, + None => { + kani::assume(false); + loop {} + } + }; + assert_eq!( + slice_dst_size_for_trailing_bytes(composite_trailing, trailing_size), + Some(expected_size) + ); + // `alloc::Layout` permits sizes which are not a multiple + // of alignment, so this represents the unpadded prefix of + // the DST through the start of its trailing slice. let field_analog = - Layout::from_size_align(field_offset, field_align.get()).unwrap(); + Layout::from_size_align(field_trailing.offset, field_align.get()).unwrap(); if let Ok((actual_composite, actual_offset)) = base_analog.extend(field_analog) { assert_eq!(actual_offset, offset); - assert_eq!(actual_composite.size(), composite_offset); + assert_eq!(actual_composite.size(), composite_trailing.offset); assert_eq!(actual_composite.align(), composite.align.get()); } else { // An error here reflects that composite of `base` @@ -2175,8 +4907,8 @@ mod proofs { #[kani::proof] #[kani::should_panic] fn prove_dst_layout_extend_dst_panics() { - let base: DstLayout = kani::any(); - let field: DstLayout = kani::any(); + let base: DstLayout = any_valid_dst_layout(); + let field: DstLayout = any_valid_dst_layout(); let packed: Option = kani::any(); if let Some(max_align) = packed { @@ -2189,39 +4921,266 @@ mod proofs { let _ = base.extend(field, packed); } + #[cfg(kani_slow)] + fn prove_dst_layout_pad_to_align_dst_invariants( + layout: DstLayout, + unpadded_trailing: TrailingSliceLayout, + padded: DstLayout, + ) { + // Calling `pad_to_align` does not alter the `DstLayout`'s alignment. + assert_eq!(padded.align, layout.align); + assert_eq!(padded.statically_shallow_unpadded, layout.statically_shallow_unpadded); + + let SizeInfo::SliceDst(padded_trailing) = padded.size_info else { + panic!("The padding of a DST layout must result in a DST layout.") + }; + assert_eq!(padded_trailing.offset, unpadded_trailing.offset); + assert_eq!(padded_trailing.elem_size, unpadded_trailing.elem_size); + + // A finalized DST formula is visibly aligned: the rounding alignment + // is at least the outer alignment, and the fixed base is a multiple of + // the outer alignment. This is the invariant which lets suffix + // validation infer alignment of the object's start from alignment of + // its end. + assert!(padded_trailing.size_rounding_align_and_phase.align() >= padded.align); + #[allow(clippy::arithmetic_side_effects)] + let padded_align_mask = padded.align.get() - 1; + assert_eq!(padded_trailing.size_base & padded_align_mask, 0); + assert_eq!(padded.pad_to_align(), padded); + let padded_phase = padded_trailing.size_rounding_align_and_phase.components().1; + let Some(padded_unrounded_min) = padded_trailing.size_base.checked_add(padded_phase) else { + unreachable!() + }; + assert!(padded_unrounded_min >= padded_trailing.offset); + } + #[kani::proof] - fn prove_dst_layout_pad_to_align() { + fn prove_dst_layout_pad_to_align_sized() { use crate::util::padding_needed_for; - let layout: DstLayout = kani::any(); + let layout: DstLayout = any_valid_dst_layout(); + let SizeInfo::Sized { size: unpadded_size } = layout.size_info else { + kani::assume(false); + loop {} + }; let padded = layout.pad_to_align(); - - // Calling `pad_to_align` does not alter the `DstLayout`'s alignment. assert_eq!(padded.align, layout.align); + let SizeInfo::Sized { size: padded_size } = padded.size_info else { + panic!("The padding of a sized layout must result in a sized layout.") + }; - if let SizeInfo::Sized { size: unpadded_size } = layout.size_info { - if let SizeInfo::Sized { size: padded_size } = padded.size_info { - // If the layout is sized, it will remain sized after padding is - // added. Its sum will be its unpadded size and the size of the - // trailing padding needed to satisfy its alignment - // requirements. - let padding = padding_needed_for(unpadded_size, layout.align); - assert_eq!(padded_size, unpadded_size + padding); - - // Prove that calling `DstLayout::pad_to_align` behaves - // identically to `Layout::pad_to_align`. - let layout_analog = - Layout::from_size_align(unpadded_size, layout.align.get()).unwrap(); - let padded_analog = layout_analog.pad_to_align(); - assert_eq!(padded_analog.align(), layout.align.get()); - assert_eq!(padded_analog.size(), padded_size); - } else { - panic!("The padding of a sized layout must result in a sized layout.") - } - } else { - // If the layout is a DST, padding cannot be statically added. - assert_eq!(padded.size_info, layout.size_info); - } + // If the layout is sized, it will remain sized after padding is added. + // Its sum will be its unpadded size and the size of the trailing + // padding needed to satisfy its alignment requirements. + let padding = padding_needed_for(unpadded_size, layout.align); + assert_eq!(padded_size, unpadded_size + padding); + assert_eq!( + padded.statically_shallow_unpadded, + layout.statically_shallow_unpadded && padding == 0 + ); + + // Prove that calling `DstLayout::pad_to_align` behaves identically to + // `Layout::pad_to_align`. + let layout_analog = Layout::from_size_align(unpadded_size, layout.align.get()).unwrap(); + let padded_analog = layout_analog.pad_to_align(); + assert_eq!(padded_analog.align(), layout.align.get()); + assert_eq!(padded_analog.size(), padded_size); + } + + #[cfg(kani_slow)] + #[kani::proof] + fn prove_dst_layout_pad_to_align_dst_inner_rounding_transition() { + use crate::util::padding_needed_for; + + let layout: DstLayout = any_valid_dst_layout(); + let SizeInfo::SliceDst(trailing) = layout.size_info else { + kani::assume(false); + loop {} + }; + kani::assume(trailing.size_rounding_align_and_phase.align() >= layout.align); + + // `any_valid_dst_layout` requires the zero-metadata layout to be valid. + // Prove that this semantic condition bounds both the padded minimum + // and the implementation intermediate; do not assume the intermediate + // merely because the implementation uses it. + let Some(min_size) = trailing.size_for_elems(0) else { unreachable!() }; + let min_padding = padding_needed_for(min_size, layout.align); + let Some(padded_min_size) = min_size.checked_add(min_padding) else { unreachable!() }; + assert!(padded_min_size <= DstLayout::MAX_SIZE); + + let padding = padding_needed_for(trailing.size_base, layout.align); + let Some(expected_base) = trailing.size_base.checked_add(padding) else { unreachable!() }; + + let padded = layout.pad_to_align(); + let SizeInfo::SliceDst(padded_trailing) = padded.size_info else { unreachable!() }; + assert_eq!(padded_trailing.size_base, expected_base); + assert_eq!( + padded_trailing.size_rounding_align_and_phase, + trailing.size_rounding_align_and_phase + ); + prove_dst_layout_pad_to_align_dst_invariants(layout, trailing, padded); + } + + #[kani::proof] + #[kani::solver(kissat)] + fn prove_dst_layout_pad_to_align_dst_inner_rounding_formula() { + use crate::util::padding_needed_for; + + let size_align = any_layout_align(); + let outer_align = any_layout_align(); + kani::assume(size_align >= outer_align); + let size_base: usize = kani::any(); + let raw_size_phase: usize = kani::any(); + #[allow(clippy::arithmetic_side_effects)] + let size_phase = raw_size_phase & (size_align.get() - 1); + let trailing_size: usize = kani::any(); + + // Evaluate `size_base + round_up(size_phase + trailing_size, + // size_align)`, then round that complete size to `outer_align`. + // Compare with the equivalent expression obtained by rounding only + // `size_base` to `outer_align`, since `size_align` is a multiple of + // `outer_align`. Valid Rust objects have representable sizes; discard + // only trailing byte counts whose input or padded object size is not + // representable. + let Some(unrounded_inner) = size_phase.checked_add(trailing_size) else { + kani::assume(false); + loop {} + }; + let Some(rounded_inner) = + unrounded_inner.checked_add(padding_needed_for(unrounded_inner, size_align)) + else { + kani::assume(false); + loop {} + }; + let Some(unpadded_size) = size_base.checked_add(rounded_inner) else { + kani::assume(false); + loop {} + }; + let Some(expected_size) = + unpadded_size.checked_add(padding_needed_for(unpadded_size, outer_align)) + else { + kani::assume(false); + loop {} + }; + + let Some(padded_base) = size_base.checked_add(padding_needed_for(size_base, outer_align)) + else { + unreachable!() + }; + let Some(actual_size) = padded_base.checked_add(rounded_inner) else { unreachable!() }; + assert_eq!(actual_size, expected_size); + } + + #[cfg(kani_slow)] + #[kani::proof] + fn prove_dst_layout_pad_to_align_dst_outer_alignment_transition() { + use crate::util::padding_needed_for; + + let layout: DstLayout = any_valid_dst_layout(); + let SizeInfo::SliceDst(trailing) = layout.size_info else { + kani::assume(false); + loop {} + }; + let size_align = trailing.size_rounding_align_and_phase.align(); + kani::assume(size_align < layout.align); + + // `any_valid_dst_layout` requires the zero-metadata layout to be valid. + // Prove that this semantic condition bounds both the padded minimum + // and every implementation intermediate; do not assume the + // intermediates merely because the implementation uses them. + let Some(min_size) = trailing.size_for_elems(0) else { unreachable!() }; + let min_padding = padding_needed_for(min_size, layout.align); + let Some(padded_min_size) = min_size.checked_add(min_padding) else { unreachable!() }; + assert!(padded_min_size <= DstLayout::MAX_SIZE); + + let base_padding = padding_needed_for(trailing.size_base, size_align); + let Some(rounded_base) = trailing.size_base.checked_add(base_padding) else { + unreachable!() + }; + let Some(t) = + rounded_base.checked_add(trailing.size_rounding_align_and_phase.components().1) + else { + unreachable!() + }; + #[allow(clippy::arithmetic_side_effects)] + let expected_phase = t & (layout.align.get() - 1); + #[allow(clippy::arithmetic_side_effects)] + let expected_base = t - expected_phase; + + let padded = layout.pad_to_align(); + let SizeInfo::SliceDst(padded_trailing) = padded.size_info else { unreachable!() }; + assert_eq!(padded_trailing.size_base, expected_base); + assert_eq!( + padded_trailing.size_rounding_align_and_phase, + RoundingAlignAndPhase::new(layout.align, expected_phase) + ); + prove_dst_layout_pad_to_align_dst_invariants(layout, trailing, padded); + } + + #[kani::proof] + #[kani::solver(kissat)] + fn prove_dst_layout_pad_to_align_dst_outer_alignment_formula() { + use crate::util::padding_needed_for; + + let size_align = any_layout_align(); + let outer_align = any_layout_align(); + kani::assume(size_align < outer_align); + let size_base: usize = kani::any(); + let raw_size_phase: usize = kani::any(); + #[allow(clippy::arithmetic_side_effects)] + let size_phase = raw_size_phase & (size_align.get() - 1); + let trailing_size: usize = kani::any(); + + // Evaluate `size_base + round_up(size_phase + trailing_size, + // size_align)`, then round that complete size to `outer_align`. + let Some(unrounded_inner) = size_phase.checked_add(trailing_size) else { + kani::assume(false); + loop {} + }; + let Some(rounded_inner) = + unrounded_inner.checked_add(padding_needed_for(unrounded_inner, size_align)) + else { + kani::assume(false); + loop {} + }; + let Some(unpadded_size) = size_base.checked_add(rounded_inner) else { + kani::assume(false); + loop {} + }; + let Some(expected_size) = + unpadded_size.checked_add(padding_needed_for(unpadded_size, outer_align)) + else { + kani::assume(false); + loop {} + }; + + // Since `outer_align` is a larger multiple of `size_align`, absorb + // the inner rounding into the outer rounding and split the fixed + // contribution into a base and phase: + // + // fixed_contribution = round_up(size_base, size_align) + size_phase + // new_phase = fixed_contribution % outer_align + // new_base = fixed_contribution - new_phase + // new_base + round_up(new_phase + trailing_size, outer_align) + let Some(rounded_base) = size_base.checked_add(padding_needed_for(size_base, size_align)) + else { + unreachable!() + }; + let Some(t) = rounded_base.checked_add(size_phase) else { unreachable!() }; + #[allow(clippy::arithmetic_side_effects)] + let new_phase = t & (outer_align.get() - 1); + #[allow(clippy::arithmetic_side_effects)] + let new_base = t - new_phase; + let Some(new_unrounded_inner) = new_phase.checked_add(trailing_size) else { + unreachable!() + }; + let Some(new_rounded_inner) = + new_unrounded_inner.checked_add(padding_needed_for(new_unrounded_inner, outer_align)) + else { + unreachable!() + }; + let Some(actual_size) = new_base.checked_add(new_rounded_inner) else { unreachable!() }; + assert_eq!(actual_size, expected_size); } } diff --git a/rust/zerocopy/src/lib.rs b/rust/zerocopy/src/lib.rs index 572f0563fe2f2e..6dbeeb7488be32 100644 --- a/rust/zerocopy/src/lib.rs +++ b/rust/zerocopy/src/lib.rs @@ -253,7 +253,7 @@ )] #![cfg_attr( __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, - deny(fuzzy_provenance_casts, lossy_provenance_casts) + deny(implicit_provenance_casts) )] #![deny( clippy::all, @@ -315,10 +315,6 @@ clippy::indexing_slicing, ))] #![cfg_attr(not(any(test, kani, feature = "std")), no_std)] -#![cfg_attr( - all(feature = "simd-nightly", target_arch = "arm"), - feature(stdarch_arm_neon_intrinsics) -)] #![cfg_attr( all(feature = "simd-nightly", any(target_arch = "powerpc", target_arch = "powerpc64")), feature(stdarch_powerpc) @@ -326,10 +322,6 @@ #![cfg_attr(feature = "float-nightly", feature(f16, f128))] #![cfg_attr(doc_cfg, feature(doc_cfg))] #![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(coverage_attribute))] -#![cfg_attr( - any(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, miri), - feature(layout_for_ptr) -)] #![cfg_attr(all(test, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), feature(test))] // This is a hack to allow zerocopy-derive derives to work in this crate. They @@ -401,7 +393,7 @@ pub use crate::{ wrappers::*, }; -#[cfg(any(feature = "alloc", test, kani))] +#[cfg(any(feature = "alloc", test))] extern crate alloc; #[cfg(any(feature = "alloc", test))] use alloc::{boxed::Box, vec::Vec}; @@ -411,7 +403,7 @@ use core::alloc::Layout; // Used by `KnownLayout`. #[doc(hidden)] pub use crate::layout::*; -// Used by `TryFromBytes::is_bit_valid`. +// Used by `TryFromBytes::is_safe`. #[doc(hidden)] pub use crate::pointer::{invariant::BecauseImmutable, Maybe, Ptr}; // For each trait polyfill, as soon as the corresponding feature is stable, the @@ -786,10 +778,16 @@ pub unsafe trait KnownLayout { /// where `size == size_of::()` /// - If `Self` is a slice DST, then `LAYOUT.size_info == /// SizeInfo::SliceDst(slice_layout)` where: + /// - `slice_layout.size_rounding_align_and_phase` decodes to an alignment + /// that is a power of two and a phase which is less than that alignment /// - The size, `size`, of an instance of `Self` with `elems` trailing - /// slice elements is equal to `slice_layout.offset + - /// slice_layout.elem_size * elems` rounded up to the nearest multiple - /// of `LAYOUT.align` + /// slice elements is equal to `slice_layout.size_base + + /// round_up(size_phase + slice_layout.elem_size * elems, size_align)`, + /// where `(size_align, size_phase)` is decoded from + /// `slice_layout.size_rounding_align_and_phase` + /// - The trailing slice begins at byte offset `slice_layout.offset` + /// - `slice_layout.offset + slice_layout.elem_size * elems` does not + /// overflow and is no greater than `size` /// - For such an instance, any bytes in the range `[slice_layout.offset + /// slice_layout.elem_size * elems, size)` are padding and must not be /// assumed to be initialized @@ -1004,11 +1002,7 @@ impl PointerMetadata for usize { #[inline] fn size_for_metadata(self, layout: DstLayout) -> Option { match layout.size_info { - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => { - let slice_len = elem_size.checked_mul(self)?; - let without_padding = offset.checked_add(slice_len)?; - without_padding.checked_add(util::padding_needed_for(without_padding, layout.align)) - } + SizeInfo::SliceDst(trailing) => trailing.size_for_elems(self), // NOTE: This branch is unreachable, but we return `None` rather // than `unreachable!()` to avoid generating panic paths. SizeInfo::Sized { .. } => None, @@ -1073,29 +1067,85 @@ unsafe impl KnownLayout for [T] { #[inline(always)] fn pointer_to_metadata(ptr: *mut [T]) -> usize { - #[allow(clippy::as_conversions)] - let slc = ptr as *const [()]; + #[cfg(not(no_zerocopy_slice_ptr_len_1_79_0))] + { + ptr.len() + } - // SAFETY: - // - `()` has alignment 1, so `slc` is trivially aligned. - // - `slc` was derived from a non-null pointer. - // - The size is 0 regardless of the length, so it is sound to - // materialize a reference regardless of location. - // - By invariant, `self.ptr` has valid provenance. - let slc = unsafe { &*slc }; - - // This is correct because the preceding `as` cast preserves the number - // of slice elements. [1] - // - // [1] Per https://doc.rust-lang.org/reference/expressions/operator-expr.html#pointer-to-pointer-cast: - // - // For slice types like `[T]` and `[U]`, the raw pointer types `*const - // [T]`, `*mut [T]`, `*const [U]`, and `*mut [U]` encode the number of - // elements in this slice. Casts between these raw pointer types - // preserve the number of elements. ... The same holds for `str` and - // any compound type whose unsized tail is a slice type, such as - // struct `Foo(i32, [u8])` or `(u64, Foo)`. - slc.len() + #[cfg(no_zerocopy_slice_ptr_len_1_79_0)] + { + // `*mut [T]::len` was not stable before Rust 1.79. In every Rust + // version from our 1.56 MSRV through 1.78, `Hash for *mut T` + // decomposes a raw pointer and passes its address and metadata to + // the hasher in that order [1]. `Hash for usize` passes its value + // to `Hasher::write_usize` [2]. Capture those two values to obtain + // a candidate for the slice length without dereferencing `ptr` or + // constructing a reference. + // + // This historical `Hash` implementation is only used to produce a + // candidate. Before returning it, we reconstruct a raw slice and + // authenticate the candidate with `ptr::eq`, which compares slice + // lengths as well as addresses [3]. Thus, an unexpected `Hash` + // implementation cannot cause us to return incorrect metadata; it + // can only fail to produce an authenticated candidate. + // + // [1] Per https://doc.rust-lang.org/1.56.0/src/core/hash/mod.rs.html#776-782: + // + // let (address, metadata) = self.to_raw_parts(); + // state.write_usize(address as usize); + // metadata.hash(state); + // + // [2] Per https://doc.rust-lang.org/1.56.0/src/core/hash/mod.rs.html#628-656: + // + // fn hash(&self, state: &mut H) { + // state.$meth(*self) + // } + // ... + // (usize, write_usize), + // + // [3] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.eq.html: + // + // Slices are also compared by their length (fat pointers). + struct MetadataHasher { + values: [usize; 2], + writes: usize, + valid: bool, + } + + impl Hasher for MetadataHasher { + #[inline(always)] + fn finish(&self) -> u64 { + 0 + } + + #[inline(always)] + fn write(&mut self, _bytes: &[u8]) { + self.valid = false; + } + + #[inline(always)] + fn write_usize(&mut self, value: usize) { + match self.values.get_mut(self.writes) { + Some(slot) => *slot = value, + None => self.valid = false, + } + self.writes = self.writes.saturating_add(1); + } + } + + let mut hasher = MetadataHasher { values: [0; 2], writes: 0, valid: true }; + core::hash::Hash::hash(&ptr, &mut hasher); + assert!( + hasher.valid && hasher.writes == 2, + "unexpected raw-pointer Hash implementation" + ); + + let elems = hasher.values[1]; + #[allow(clippy::as_conversions)] + let reconstructed = ptr::slice_from_raw_parts_mut(ptr as *mut T, elems); + assert!(ptr::eq(ptr, reconstructed), "captured value is not raw-slice metadata"); + elems + } } } @@ -1182,11 +1232,29 @@ pub const UNION_VARIANT_ID: i128 = -2; #[doc(hidden)] pub const REPR_C_UNION_VARIANT_ID: i128 = -3; +/// Marker types used to disambiguate implementations of projection traits that +/// would otherwise conflict. +#[doc(hidden)] +#[allow(missing_copy_implementations, missing_debug_implementations)] +pub mod project_clients { + pub enum TryFromBytesDerive {} + + pub enum ProjectDerive {} +} + +#[cfg(any(feature = "derive", test))] +#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))] +#[doc(hidden)] +pub use zerocopy_derive::Project; + /// # Safety /// -/// `Self::ProjectToTag` must satisfy its safety invariant. +/// `>::ProjectToTag` must satisfy its safety invariant. +/// +/// The `Client` parameter exists solely to disambiguate between implementations +/// of `HasTag` that would otherwise conflict. #[doc(hidden)] -pub unsafe trait HasTag { +pub unsafe trait HasTag { fn only_derive_is_allowed_to_implement_this_trait() where Self: Sized; @@ -1198,28 +1266,35 @@ pub unsafe trait HasTag { /// /// # Safety /// - /// It must be the case that, for all `slf: Ptr<'_, Self, I>`, it is sound - /// to project from `slf` to `Ptr<'_, Self::Tag, I>` using this projection. + /// It must be the case that, for all `slf: Ptr<'_, Self, I>` where + /// `I::Aliasing` is `Shared`, it is sound to project it using this + /// projection to a `Ptr<'_, Self::Tag, (Shared, I::Alignment, + /// I::Validity)>`. type ProjectToTag: pointer::cast::Project; } /// Projects a given field from `Self`. /// -/// All implementations of `HasField` for a particular field `f` in `Self` -/// should use the same `Field` type; this ensures that `Field` is inferable -/// given an explicit `VARIANT_ID` and `FIELD_ID`. +/// All implementations of `HasField` for a particular `Client` and field `f` +/// in `Self` should use the same `Field` type; this ensures that `Field` is +/// inferable given an explicit `Client`, `VARIANT_ID`, and `FIELD_ID`. +/// +/// The `Client` parameter exists solely to disambiguate between implementations +/// of `HasField` that would otherwise conflict. /// /// # Safety /// /// A field `f` is `HasField` for `Self` if and only if: /// -/// - If `Self` has the layout of a struct or union type, then `VARIANT_ID` is -/// `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID` respectively; otherwise, if -/// `Self` has the layout of an enum type, `VARIANT_ID` is the numerical index -/// of the enum variant in which `f` appears. Note that `Self` does not need -/// to actually *be* such a type – it just needs to have the same layout as -/// such a type. For example, a `#[repr(transparent)]` wrapper around an enum -/// has the same layout as that enum. +/// - If `Self` has the layout of a struct type, `VARIANT_ID` is +/// `STRUCT_VARIANT_ID`. If `Self` has the layout of a `repr(C)` union type, +/// `VARIANT_ID` is `REPR_C_UNION_VARIANT_ID`; for other union layouts, it is +/// `UNION_VARIANT_ID`. Otherwise, if `Self` has the layout of an enum type and +/// `f` appears in a variant named `v`, `VARIANT_ID` is +/// `zerocopy::ident_id!(v)`. Note that `Self` does not need to actually *be* +/// such a type – it just needs to have the same layout as such a type. For +/// example, a `#[repr(transparent)]` wrapper around an enum has the same +/// layout as that enum. /// - If `f` has name `n`, `FIELD_ID` is `zerocopy::ident_id!(n)`; otherwise, /// if `f` is at index `i`, `FIELD_ID` is `zerocopy::ident_id!(i)`. /// - `Field` is a type with the same visibility as `f`. @@ -1232,8 +1307,8 @@ pub unsafe trait HasTag { /// /// The implementation of `project` must satisfy its safety post-condition. #[doc(hidden)] -pub unsafe trait HasField: - HasTag +pub unsafe trait HasField: + HasTag { fn only_derive_is_allowed_to_implement_this_trait() where @@ -1263,15 +1338,20 @@ pub unsafe trait HasField: /// other words, it is a type-level function over invariants; `I` goes in, /// `Self::Invariants` comes out. /// +/// The `Client` parameter exists solely to disambiguate between implementations +/// of `ProjectField` (and their corresponding `HasField` implementations) that +/// would otherwise conflict. +/// /// # Safety /// -/// `T: ProjectField` if, for a -/// `ptr: Ptr<'_, T, I>` such that `T::is_projectable(ptr).is_ok()`, -/// `>::project(ptr.as_inner())` +/// `T: ProjectField` if, for a +/// `ptr: Ptr<'_, T, I>` such that `T::is_projectable` returns `Ok(())` +/// when passed the tag pointer projected from `ptr`, +/// `>::project(ptr.as_inner())` /// conforms to `T::Invariants`. #[doc(hidden)] -pub unsafe trait ProjectField: - HasField +pub unsafe trait ProjectField: + HasField where I: invariant::Invariants, { @@ -1290,29 +1370,35 @@ where /// Is the given field projectable from `ptr`? /// - /// If a field with [`Self::Invariants`] is projectable from the referent, - /// this function produces an `Ok(ptr)` from which the projection can be - /// made; otherwise `Err`. + /// If a field with [`Self::Invariants`] is projectable from the containing + /// value whose projected tag is `ptr`, this function produces `Ok(())`; + /// otherwise it produces `Err`. /// /// This method must be overriden if the field's projectability depends on /// the value of the bytes in `ptr`. #[inline(always)] - fn is_projectable<'a>(_ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> { + fn is_projectable<'a>( + _ptr: Ptr< + 'a, + >::Tag, + (invariant::Shared, I::Alignment, I::Validity), + >, + ) -> Result<(), Self::Error> { trait IsInfallible { const IS_INFALLIBLE: bool; } - struct Projection( - PhantomData<(Field, I, T)>, + struct Projection( + PhantomData<(Client, Field, I, T)>, ) where - T: ?Sized + HasField, + T: ?Sized + HasField, I: invariant::Invariants; - impl IsInfallible - for Projection + impl IsInfallible + for Projection where - T: ?Sized + HasField, + T: ?Sized + HasField, I: invariant::Invariants, { const IS_INFALLIBLE: bool = { @@ -1323,7 +1409,9 @@ where // referent. This default implementation of `is_projectable` // is non-destructive, as it does not overwrite any part of // the referent. - crate::STRUCT_VARIANT_ID | crate::UNION_VARIANT_ID => true, + crate::STRUCT_VARIANT_ID + | crate::UNION_VARIANT_ID + | crate::REPR_C_UNION_VARIANT_ID => true, _enum_variant => { use crate::invariant::{Validity, ValidityKind}; match I::Validity::KIND { @@ -1336,9 +1424,9 @@ where // type – the type itself is irrelevant. ValidityKind::Uninit | ValidityKind::Initialized => true, // The projectability of an enum field from an - // `AsInitialized` or `Valid` state is a dynamic + // `AsInitialized` or `Safe` state is a dynamic // property of its tag. - ValidityKind::AsInitialized | ValidityKind::Valid => false, + ValidityKind::AsInitialized | ValidityKind::Safe => false, } } }; @@ -1348,7 +1436,7 @@ where } const_assert!( - as IsInfallible>::IS_INFALLIBLE + as IsInfallible>::IS_INFALLIBLE ); Ok(()) @@ -1614,6 +1702,64 @@ pub unsafe trait Immutable { /// } /// ``` /// +#[cfg_attr( + zerocopy_unstable_ptr, + doc = r#" +# Field invariants + +This experimental feature requires `--cfg zerocopy_unstable_ptr`. + +Named fields of structs, enum variants, and unions can specify additional +runtime checks using `#[zerocopy(invariant(expression))]`: + +``` +# use zerocopy_derive::TryFromBytes; +#[derive(TryFromBytes)] +struct Foo { + a: u8, + #[zerocopy(invariant((*a.read() % 2) == (*b.read() as u8)))] + b: bool, + #[zerocopy(invariant(*c.read() > 0))] + c: i16, +} +``` + +Each expression must return a `bool`. It has access to validated, read-only +[`Ptr`]s to the current field and all preceding fields of the struct or +variant, using their field names. A union's invariants have access only to +the current field. The expression can use the existing [`Ptr`] APIs to +inspect those fields. In this example, `read()` copies each field without +requiring alignment, and dereferencing the resulting [`ReadOnly`] accesses +the copied value. + +Rust's usual restrictions on local bindings apply; for example, a field name +cannot shadow an in-scope constant. + +Fields are checked in declaration order. Each field's bit validity is +checked before its invariants run. Multiple invariants on a field run in +attribute order. For structs and enums, validation stops at the first invalid +field or invariant that returns `false`. For unions, a failed bit-validity +check or invariant causes validation to try the next field; validation +succeeds as soon as one field and all its invariants pass. Each expression +runs in its own closure; `return` returns from that expression, and panics +propagate to the caller. Only the selected enum variant's fields and +invariants are checked. Expressions may have arbitrary side effects, even +when the conversion fails or its result is discarded. + +These predicates are additional acceptance checks beyond Rust's bit validity; +bit-valid bytes may still be rejected. See [What is a "valid instance"?] for +the distinction. + +[What is a "valid instance"?]: trait@TryFromBytes#what-is-a-valid-instance + +Invariants are not supported on tuple fields. Types with invariants cannot +derive [`FromZeros`] or [`FromBytes`], whose conversions do not perform runtime +validation. These checks apply to conversions through [`TryFromBytes`]; they +do not restrict ordinary construction or mutation of Rust values. + +"# +)] +/// /// # Portability /// /// To ensure consistent endianness for enums with multi-byte representations, @@ -1708,6 +1854,14 @@ pub use zerocopy_derive::TryFromBytes; /// If you are negatively affected by lack of support for a particular type, /// we encourage you to let us know by [filing an issue][github-repo]. /// +/// In this trait's conversion methods, a "valid instance" must also pass any +/// configured field invariants, including those on nested fields. Rust bit +/// validity alone does not guarantee that a conversion succeeds: an invariant +/// may reject otherwise bit-valid bytes. Such predicates check acceptance at +/// conversion time; they do not constrain subsequent mutation or ordinary Rust +/// construction. See the [derive's field invariants][derive] documentation for +/// the experimental attribute's syntax, evaluation order, and side effects. +/// /// # `TryFromBytes` is not symmetrical with [`IntoBytes`] /// /// There are some types which implement both `TryFromBytes` and [`IntoBytes`], @@ -1754,25 +1908,25 @@ pub unsafe trait TryFromBytes { /// /// # Safety /// - /// Unsafe code may assume that, if `is_bit_valid(candidate)` returns true, + /// Unsafe code may assume that, if `is_safe(candidate)` returns true, /// `*candidate` contains a valid `Self`. /// /// # Panics /// - /// `is_bit_valid` may panic. Callers are responsible for ensuring that any - /// `unsafe` code remains sound even in the face of `is_bit_valid` - /// panicking. (We support user-defined validation routines; so long as - /// these routines are not required to be `unsafe`, there is no way to - /// ensure that these do not generate panics.) + /// `is_safe` may panic. Callers are responsible for ensuring that any + /// `unsafe` code remains sound even in the face of `is_safe` panicking. (We + /// support user-defined validation routines; so long as these routines are + /// not required to be `unsafe`, there is no way to ensure that these do not + /// generate panics.) /// - /// Besides user-defined validation routines panicking, `is_bit_valid` will - /// either panic or fail to compile if called on a pointer with [`Shared`] - /// aliasing when `Self: !Immutable`. + /// Besides user-defined validation routines panicking, `is_safe` will either + /// panic or fail to compile if called on a pointer with [`Shared`] aliasing + /// when `Self: !Immutable`. /// /// [`UnsafeCell`]: core::cell::UnsafeCell /// [`Shared`]: invariant::Shared #[doc(hidden)] - fn is_bit_valid(candidate: Maybe<'_, Self, A>) -> bool + fn is_safe(candidate: Maybe<'_, Self, A>) -> bool where A: invariant::Alignment; @@ -1872,7 +2026,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_bytes(source: &[u8]) -> Result<&Self, TryCastError<&[u8], Self>> @@ -1885,7 +2039,7 @@ pub unsafe trait TryFromBytes { // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to // fix before returning. - match source.try_into_valid() { + match source.try_into_safe() { Ok(valid) => Ok(valid.as_ref()), Err(e) => { Err(e.map_src(|src| src.as_bytes::().as_ref()).into()) @@ -1995,7 +2149,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_prefix(source: &[u8]) -> Result<(&Self, &[u8]), TryCastError<&[u8], Self>> @@ -2105,7 +2259,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_suffix(source: &[u8]) -> Result<(&[u8], &Self), TryCastError<&[u8], Self>> @@ -2199,7 +2353,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_bytes")] /// /// See [`TryFromBytes::try_ref_from_bytes`](#method.try_ref_from_bytes.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_bytes(bytes: &mut [u8]) -> Result<&mut Self, TryCastError<&mut [u8], Self>> @@ -2212,7 +2366,7 @@ pub unsafe trait TryFromBytes { // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to // fix before returning. - match source.try_into_valid() { + match source.try_into_safe() { Ok(source) => Ok(source.as_mut()), Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()), } @@ -2308,7 +2462,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_prefix")] /// /// See [`TryFromBytes::try_ref_from_prefix`](#method.try_ref_from_prefix.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_prefix( @@ -2408,7 +2562,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_suffix")] /// /// See [`TryFromBytes::try_ref_from_suffix`](#method.try_ref_from_suffix.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_suffix( @@ -2513,7 +2667,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_bytes_with_elems( @@ -2529,7 +2683,7 @@ pub unsafe trait TryFromBytes { // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to // fix before returning. - match source.try_into_valid() { + match source.try_into_safe() { Ok(source) => Ok(source.as_ref()), Err(e) => { Err(e.map_src(|src| src.as_bytes::().as_ref()).into()) @@ -2634,7 +2788,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_prefix_with_elems( @@ -2742,7 +2896,7 @@ pub unsafe trait TryFromBytes { @variant "dynamic_padding" ] )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_ref_from_suffix_with_elems( @@ -2838,7 +2992,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_bytes_with_elems")] /// /// See [`TryFromBytes::try_ref_from_bytes_with_elems`](#method.try_ref_from_bytes_with_elems.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_bytes_with_elems( @@ -2854,7 +3008,7 @@ pub unsafe trait TryFromBytes { // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to // fix before returning. - match source.try_into_valid() { + match source.try_into_safe() { Ok(source) => Ok(source.as_mut()), Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()), } @@ -2949,7 +3103,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_prefix_with_elems")] /// /// See [`TryFromBytes::try_ref_from_prefix_with_elems`](#method.try_ref_from_prefix_with_elems.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_prefix_with_elems( @@ -3049,7 +3203,7 @@ pub unsafe trait TryFromBytes { #[doc = codegen_header!("h5", "try_mut_from_suffix_with_elems")] /// /// See [`TryFromBytes::try_ref_from_suffix_with_elems`](#method.try_ref_from_suffix_with_elems.codegen). - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_mut_from_suffix_with_elems( @@ -3116,24 +3270,14 @@ pub unsafe trait TryFromBytes { bench = "try_read_from_bytes", format = "coco_static_size", )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_read_from_bytes(source: &[u8]) -> Result> where Self: Sized, { - // FIXME(#2981): If `align_of::() == 1`, validate `source` in-place. - - let candidate = match CoreMaybeUninit::::read_from_bytes(source) { - Ok(candidate) => candidate, - Err(e) => { - return Err(TryReadError::Size(e.with_dst())); - } - }; - // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of - // its bytes are initialized. - unsafe { try_read_from(source, candidate) } + try_read_from(source) } /// Attempts to read a `Self` from the prefix of the given `source`. @@ -3194,24 +3338,24 @@ pub unsafe trait TryFromBytes { bench = "try_read_from_prefix", format = "coco_static_size", )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_read_from_prefix(source: &[u8]) -> Result<(Self, &[u8]), TryReadError<&[u8], Self>> where Self: Sized, { - // FIXME(#2981): If `align_of::() == 1`, validate `source` in-place. - - let (candidate, suffix) = match CoreMaybeUninit::::read_from_prefix(source) { - Ok(candidate) => candidate, - Err(e) => { - return Err(TryReadError::Size(e.with_dst())); - } + let (prefix, suffix) = match SplitAt::split_at(source, mem::size_of::()) { + Some(split) => split.via_immutable(), + None => return Err(SizeError::new(source).into()), }; - // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of - // its bytes are initialized. - unsafe { try_read_from(source, candidate).map(|slf| (slf, suffix)) } + match try_read_from(prefix) { + Ok(slf) => Ok((slf, suffix)), + Err(e) => Err(e.map_src( + #[inline(always)] + |_| source, + )), + } } /// Attempts to read a `Self` from the suffix of the given `source`. @@ -3273,44 +3417,72 @@ pub unsafe trait TryFromBytes { bench = "try_read_from_suffix", format = "coco_static_size", )] - #[must_use = "has no side effects"] + #[must_use = "the conversion result must be checked"] #[cfg_attr(zerocopy_inline_always, inline(always))] #[cfg_attr(not(zerocopy_inline_always), inline)] fn try_read_from_suffix(source: &[u8]) -> Result<(&[u8], Self), TryReadError<&[u8], Self>> where Self: Sized, { - // FIXME(#2981): If `align_of::() == 1`, validate `source` in-place. - - let (prefix, candidate) = match CoreMaybeUninit::::read_from_suffix(source) { - Ok(candidate) => candidate, - Err(e) => { - return Err(TryReadError::Size(e.with_dst())); - } + let split_at = match source.len().checked_sub(mem::size_of::()) { + Some(split_at) => split_at, + None => return Err(SizeError::new(source).into()), }; - // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of - // its bytes are initialized. - unsafe { try_read_from(source, candidate).map(|slf| (prefix, slf)) } + // SAFETY: `checked_sub` returned the difference without overflow [1], + // so `split_at = source.len() - size_of::() <= source.len()`. + // This satisfies `SplitAt::split_at_unchecked`'s precondition. + // + // [1] Per https://doc.rust-lang.org/1.56.0/std/primitive.usize.html#method.checked_sub: + // + // Checked integer subtraction. Computes `self - rhs`, returning + // `None` if overflow occurred. + let (prefix, suffix) = + unsafe { SplitAt::split_at_unchecked(source, split_at) }.via_immutable(); + match try_read_from(suffix) { + Ok(slf) => Ok((prefix, slf)), + Err(e) => Err(e.map_src( + #[inline(always)] + |_| source, + )), + } } } +/// Validates and interprets the given affix of `source`'s bytes as a `&T`. +/// +/// Returns the destination reference and excess bytes on success. All errors, +/// including validity errors, contain the original `&S`. #[inline(always)] -fn try_ref_from_prefix_suffix( - source: &[u8], +fn try_ref_from_prefix_suffix( + source: &S, cast_type: CastType, meta: Option, -) -> Result<(&T, &[u8]), TryCastError<&[u8], T>> { - match Ptr::from_ref(source).try_cast_into::(cast_type, meta) { - Ok((source, prefix_suffix)) => { +) -> Result<(&T, &[u8]), TryCastError<&S, T>> +where + S: IntoBytes + Immutable + ?Sized, + T: TryFromBytes + KnownLayout + Immutable + ?Sized, +{ + match Ptr::from_ref(source.as_bytes()).try_cast_into::(cast_type, meta) { + Ok((candidate, prefix_suffix)) => { // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to // fix before returning. - match source.try_into_valid() { + match candidate.try_into_safe() { Ok(valid) => Ok((valid.as_ref(), prefix_suffix.as_ref())), - Err(e) => Err(e.map_src(|src| src.as_bytes::().as_ref()).into()), + Err(e) => Err(e + .map_src( + #[inline(always)] + |_| source, + ) + .into()), } } - Err(e) => Err(e.map_src(Ptr::as_ref).into()), + Err(e) => Err(e + .map_src( + #[inline(always)] + |_| source, + ) + .into()), } } @@ -3325,7 +3497,7 @@ fn try_mut_from_prefix_suffix Ok((valid.as_mut(), prefix_suffix.as_mut())), Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()), } @@ -3339,49 +3511,112 @@ fn swap((t, u): (T, U)) -> (U, T) { (u, t) } -/// # Safety -/// -/// All bytes of `candidate` must be initialized. #[inline(always)] -unsafe fn try_read_from( - source: S, - mut candidate: CoreMaybeUninit, -) -> Result> { +fn try_read_from(source: &S) -> Result> +where + S: IntoBytes + Immutable + ?Sized, + T: TryFromBytes, +{ + let bytes = source.as_bytes(); + + if bytes.len() != mem::size_of::() { + return Err(SizeError::new(source).into()); + } + + // FIXME(#2981): Avoid the validation copy when validation can safely use + // a pointer derived from `source`'s shared borrow. + + // Initialize the candidate in its final location. A typed move of a + // `MaybeUninit` may discard initialized bytes at `T`'s padding offsets + // [1], but validation requires every byte to be initialized. Do not move + // `candidate` between this copy and validation. + // + // [1] Per https://doc.rust-lang.org/1.93.1/std/mem/union.MaybeUninit.html#validity: + // + // Moving or copying a value of type `MaybeUninit` (i.e., performing a + // "typed copy") will exactly preserve the contents, including the + // provenance, of all non-padding bytes of type `T` in the value's + // representation. + let mut candidate = CoreMaybeUninit::::uninit(); + + // SAFETY: The earlier `if bytes.len() != mem::size_of::()` returns on a + // size mismatch, so reaching this copy implies that `bytes.len()` equals + // `mem::size_of::()`. Copying that many `u8`s cannot overrun `bytes`. + // `candidate.as_mut_ptr()` is writable for the same number of bytes because + // `MaybeUninit` has `T`'s size. Both pointers are non-null and aligned + // for `u8`, including when `T` is zero-sized. The fresh local allocation + // cannot overlap `source`, which is an argument. Thus the read, write, + // alignment, and non-overlap requirements of [2] hold. Copying `u8`s + // initializes every destination byte, including those at `T`'s padding + // offsets. + // + // These writes cannot violate `candidate`'s bit validity because every bit + // pattern is valid for `MaybeUninit` [3], even if it is invalid for `T`. + // + // [2] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.copy_nonoverlapping.html: + // + // Copies `count * size_of::()` bytes from `src` to `dst`. The source + // and destination must *not* overlap. + // + // [3] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#layout: + // + // ... any bit value is valid for a `MaybeUninit` ... + unsafe { + ptr::copy_nonoverlapping( + bytes.as_ptr(), + candidate.as_mut_ptr().cast::(), + mem::size_of::(), + ); + } + // We use `from_mut` despite not mutating via `c_ptr` so that we don't need // to add a `T: Immutable` bound. let c_ptr = Ptr::from_mut(&mut candidate); + // SAFETY: `c_ptr` has no uninitialized sub-ranges because it derived from - // `candidate`, which the caller promises is entirely initialized. Since - // `candidate` is a `MaybeUninit`, it has no validity requirements, and so - // no values written to an `Initialized` `c_ptr` can violate its validity. - // Since `c_ptr` has `Exclusive` aliasing, no mutations may happen except - // via `c_ptr` so long as it is live, so we don't need to worry about the - // fact that `c_ptr` may have more restricted validity than `candidate`. + // `candidate`, whose bytes were all initialized by the copy above and which + // has not been moved since. Since `candidate` is a `MaybeUninit`, it has no + // validity requirements, and so no values written to an `Initialized` + // `c_ptr` can violate its validity. Since `c_ptr` has `Exclusive` aliasing, + // no mutations may happen except via `c_ptr` so long as it is live, so we + // don't need to worry about the fact that `c_ptr` may have more restricted + // validity than `candidate`. let c_ptr = unsafe { c_ptr.assume_validity::() }; - let mut c_ptr = c_ptr.cast::<_, crate::pointer::cast::CastSized, _>(); - // Since we don't have `T: KnownLayout`, we hack around that by using - // `Wrapping`, which implements `KnownLayout` even if `T` doesn't. + let c_ptr = c_ptr.cast::<_, crate::pointer::cast::CastSized, _>(); + + // SAFETY: `c_ptr` originated from a reference to `candidate`, so its + // address is aligned for `MaybeUninit`, which has `T`'s alignment [1]. + // `CastSized` preserves that address. `ReadOnly` is `repr(transparent)` + // with a single `T` field, so it also has `T`'s alignment [2], including + // when `T` is zero-sized. // + // [1] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#layout: + // + // `MaybeUninit` is guaranteed to have the same size, alignment, and + // ABI as `T`: + // + // [2] Per https://doc.rust-lang.org/1.93.1/reference/type-layout.html#the-transparent-representation: + // + // ... same layout and ABI as the only non-size 0 non-alignment 1 field, + // if present, or unit otherwise. + let mut c_ptr = unsafe { c_ptr.assume_alignment::() }; + // This call may panic. If that happens, it doesn't cause any soundness // issues, as we have not generated any invalid state which we need to fix // before returning. - if !Wrapping::::is_bit_valid(c_ptr.reborrow_shared().forget_aligned()) { + if !T::is_safe(c_ptr.reborrow_shared()) { return Err(ValidityError::new(source).into()); } - fn _assert_same_size_and_validity() - where - Wrapping: pointer::TransmuteFrom, - T: pointer::TransmuteFrom, invariant::Valid, invariant::Valid>, - { - } - - _assert_same_size_and_validity::(); - - // SAFETY: We just validated that `candidate` contains a valid - // `Wrapping`, which has the same size and bit validity as `T`, as - // guaranteed by the preceding type assertion. + // SAFETY: `T::is_safe` returned true for `candidate`'s initialized bytes, + // so it contains a valid `T`, as required by [1]. Validation used a shared + // `ReadOnly` pointer, and the bytes have not been modified since. + // + // [1] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#method.assume_init: + // + // It is up to the caller to guarantee that the `MaybeUninit` really is + // in an initialized state. Ok(unsafe { candidate.assume_init() }) } @@ -5464,22 +5699,32 @@ pub unsafe trait FromBytes: FromZeros { } } -/// Interprets the given affix of the given bytes as a `&Self`. +/// Interprets the given affix of `source`'s bytes as a `&T`. /// -/// This method computes the largest possible size of `Self` that can fit in the -/// prefix or suffix bytes of `source`, then attempts to return both a reference -/// to those bytes interpreted as a `Self`, and a reference to the excess bytes. -/// If there are insufficient bytes, or if that affix of `source` is not -/// appropriately aligned, this returns `Err`. +/// This method uses `meta` if provided; otherwise, it computes the largest +/// possible size of `T` that can fit in the prefix or suffix bytes of `source`. +/// It returns both a reference to those bytes interpreted as a `T`, and a +/// reference to the excess bytes. If there are insufficient bytes, or if that +/// affix of `source` is not appropriately aligned, this returns `Err` containing +/// the original `&S`. #[inline(always)] -fn ref_from_prefix_suffix( - source: &[u8], +fn ref_from_prefix_suffix( + source: &S, meta: Option, cast_type: CastType, -) -> Result<(&T, &[u8]), CastError<&[u8], T>> { - let (slf, prefix_suffix) = Ptr::from_ref(source) +) -> Result<(&T, &[u8]), CastError<&S, T>> +where + S: IntoBytes + Immutable + ?Sized, + T: FromBytes + KnownLayout + Immutable + ?Sized, +{ + let (slf, prefix_suffix) = Ptr::from_ref(source.as_bytes()) .try_cast_into::<_, BecauseImmutable>(cast_type, meta) - .map_err(|err| err.map_src(|s| s.as_ref()))?; + .map_err(|err| { + err.map_src( + #[inline(always)] + |_| source, + ) + })?; Ok((slf.recall_validity().as_ref(), prefix_suffix.as_ref())) } @@ -5602,6 +5847,10 @@ fn mut_from_prefix_suffix( /// if its field is [`IntoBytes`]; else, /// - if the type has no generic parameters, it is [`IntoBytes`] if the type /// is sized and has no padding bytes; else, +/// - if the type is `repr(C)` without an `align(N)` modifier for `N > 1` +/// (it may have `align(1)` or `packed(N)`), and every field is `T`, +/// `[T; N]`, or a final `[T]` for the same type parameter `T`, it is +/// [`IntoBytes`]; else, /// - if the type is `repr(C)`, its fields must be [`Unaligned`]. /// - If the type is an enum: /// - It must have a defined representation (`repr`s `C`, `u8`, `u16`, `u32`, @@ -6493,6 +6742,39 @@ mod tests { } } + // These types exercise a nested slice DST whose inner trailing-padding + // operation cannot be flattened into the alignment of the packed outer + // type. For `NestedPackedDstOuter<[u8]>`, the trailing slice begins at + // offset 7, but the size for `elems` elements is + // `2 + round_up(5 + elems, 4)`. + // + // The Reference rejects directly placing an alignment-4 field in a + // `repr(packed(2))` type. Rust currently accepts this construction through + // generic indirection. So long as Rust accepts it, this test deliberately + // treats its layout as following the otherwise-applicable `repr(C)` and + // `repr(packed)` rules. + #[derive(FromBytes, Immutable, KnownLayout)] + #[repr(C, align(4))] + struct NestedPackedDstWord([u8; 4]); + + #[derive(FromBytes, Immutable, KnownLayout)] + #[repr(C)] + struct NestedPackedDstInner { + a: NestedPackedDstWord, + b: u8, + tail: T, + } + + #[derive(FromBytes, Immutable, KnownLayout)] + #[repr(C, packed(2))] + struct NestedPackedDstOuter { + z: u8, + inner: ManuallyDrop>, + } + + #[repr(C, align(2))] + struct Align2(T); + #[test] fn test_known_layout() { // Test that `$ty` and `ManuallyDrop<$ty>` have the expected layout. @@ -6507,17 +6789,25 @@ mod tests { }; } - let layout = - |offset, align, trailing_slice_elem_size, statically_shallow_unpadded| DstLayout { - align: NonZeroUsize::new(align).unwrap(), + let layout = |offset, align, trailing_slice_elem_size, statically_shallow_unpadded| { + let align = NonZeroUsize::new(align).unwrap(); + let size_phase = offset % align; + DstLayout { + align, size_info: match trailing_slice_elem_size { None => SizeInfo::Sized { size: offset }, - Some(elem_size) => { - SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) - } + Some(elem_size) => SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base: offset - size_phase, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + align, size_phase, + ), + }), }, statically_shallow_unpadded, - }; + } + }; test!((), layout(0, 1, None, false)); test!(u8, layout(1, 1, None, false)); @@ -6533,6 +6823,153 @@ mod tests { test!(str, layout(0, 1, Some(1), true)); } + #[test] + fn test_known_layout_pointer_to_metadata() { + fn test(data: *mut T, elems: usize) { + let ptr = ptr::slice_from_raw_parts_mut(data, elems); + assert_eq!(<[T] as KnownLayout>::pointer_to_metadata(ptr), elems); + } + + for elems in [0, 1, usize::MAX] { + test(ptr::null_mut::(), elems); + } + + test(NonNull::::dangling().as_ptr(), 0); + + let mut bytes = [1, 2, 3]; + test(bytes.as_mut_ptr(), bytes.len()); + assert_eq!(bytes, [1, 2, 3]); + + test(NonNull::<()>::dangling().as_ptr(), usize::MAX); + + // Retaining and inspecting a raw pointer after freeing its allocation + // is safe so long as the pointer is not dereferenced. + let deallocated = { + let mut byte = Box::new(0u8); + let ptr: *mut u8 = &mut *byte; + drop(byte); + ptr + }; + test(deallocated, 3); + + // Metadata extraction requires neither element storage nor alignment + // for `T`; it does not access the pointer's referent. + test(NonNull::::dangling().as_ptr().cast::(), usize::MAX); + } + + #[cfg(feature = "derive")] + #[test] + fn test_known_layout_pointer_to_metadata_derive() { + #[derive(KnownLayout)] + #[repr(C)] + struct Dst { + prefix: u8, + trailing: [u8], + } + + for elems in [0, 1, usize::MAX] { + let ptr = ptr::slice_from_raw_parts_mut(ptr::null_mut::(), elems); + #[allow(clippy::as_conversions)] + let ptr = ptr as *mut Dst; + assert_eq!(Dst::pointer_to_metadata(ptr), elems); + } + } + + #[test] + fn test_nested_packed_dst_layout() { + fn assert_native_size(expected: usize) { + let value = NestedPackedDstOuter { + z: 0, + inner: ManuallyDrop::new(NestedPackedDstInner { + a: NestedPackedDstWord([0; 4]), + b: 0, + tail: [0; N], + }), + }; + let value: &NestedPackedDstOuter<[u8]> = &value; + assert_eq!(mem::size_of_val(value), expected); + assert_eq!(NestedPackedDstOuter::<[u8]>::size_for_metadata(N), Some(expected)); + } + + assert_native_size::<0>(10); + assert_native_size::<1>(10); + assert_native_size::<2>(10); + assert_native_size::<3>(10); + assert_native_size::<4>(14); + assert_native_size::<7>(14); + assert_native_size::<8>(18); + + let layout = NestedPackedDstOuter::<[u8]>::LAYOUT; + assert_eq!(layout.align.get(), 2); + assert!(layout.requires_dynamic_padding()); + assert_eq!( + layout.size_info, + SizeInfo::SliceDst(TrailingSliceLayout { + offset: 7, + elem_size: 1, + size_base: 6, + size_rounding_align_and_phase: RoundingAlignAndPhase::new( + NonZeroUsize::new(4).unwrap(), + 1 + ), + }) + ); + + // Eight bytes used to be incorrectly accepted for a zero-element + // value. The actual Rust layout requires ten bytes. + let short = Align2([0u8; 8]); + assert!(NestedPackedDstOuter::<[u8]>::ref_from_bytes_with_elems(&short.0, 0).is_err()); + assert!(pointer::Ptr::from_ref(&short.0[..]) + .try_cast_into::, pointer::BecauseImmutable>( + CastType::Prefix, + Some(0), + ) + .is_err()); + + let full = Align2([0, 0, 0, 0, 0, 0, 0, 0, b'S', b'!']); + let value = NestedPackedDstOuter::<[u8]>::ref_from_bytes_with_elems(&full.0, 0).unwrap(); + assert_eq!(mem::size_of_val(value), 10); + + // When metadata is inferred, lengths 0 through 3 all produce a + // ten-byte object. Parsing chooses the largest trailing-slice length + // which fits exactly. + let inferred = NestedPackedDstOuter::<[u8]>::ref_from_bytes(&full.0).unwrap(); + assert_eq!(pointer::Ptr::from_ref(inferred).len(), 3); + + // Trailing padding includes the inner type's padding, even when the + // outer type's alignment is reduced by packing. + let trailing_layout = trailing_slice_layout::>(); + for (elems, expected) in [(0, 3), (1, 2), (2, 1), (3, 0), (4, 3)] { + assert_eq!(trailing_layout.padding_for_elems(elems), expected); + } + } + + // Before Rust 1.76, rust-lang/rust#118537 caused `addr_of_mut!` to + // miscompute the offset of an unsized field in a packed struct. The fix is + // rust-lang/rust#118540. Keep the layout and parsing regression above on + // all supported compilers, and gate only the projection regression which + // necessarily exercises that independent compiler bug. + #[rustversion::since(1.76.0)] + #[test] + fn test_nested_packed_dst_projection() { + let full = Align2([0, 0, 0, 0, 0, 0, 0, 0, b'S', b'!']); + + // The projected inner value occupies the final eight bytes of the full + // ten-byte input, including the two sentinel bytes. + let (value, rest) = pointer::Ptr::from_ref(&full.0[..]) + .try_cast_into::, pointer::BecauseImmutable>( + CastType::Prefix, + Some(0), + ) + .unwrap(); + assert!(rest.as_ref().is_empty()); + let inner = value.project::().unwrap(); + assert_eq!( + inner.as_bytes::().as_ref(), + &[0, 0, 0, 0, 0, 0, b'S', b'!'] + ); + } + #[cfg(feature = "derive")] #[test] fn test_known_layout_derive() { @@ -6588,10 +7025,19 @@ mod tests { statically_shallow_unpadded: false, }; - let unsized_layout = |align, elem_size, offset, statically_shallow_unpadded| DstLayout { - align: NonZeroUsize::new(align).unwrap(), - size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }), - statically_shallow_unpadded, + let unsized_layout = |align, elem_size, offset, statically_shallow_unpadded| { + let align = NonZeroUsize::new(align).unwrap(); + let size_phase = offset % align; + DstLayout { + align, + size_info: SizeInfo::SliceDst(TrailingSliceLayout { + offset, + elem_size, + size_base: offset - size_phase, + size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, size_phase), + }), + statically_shallow_unpadded, + } }; // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name | @@ -7152,6 +7598,112 @@ mod tests { ); } + #[test] + fn test_try_read_from_typed_source() { + #[derive(IntoBytes, Immutable)] + #[repr(transparent)] + struct Source(T); + + fn check(source: &S, valid: bool) { + match try_read_from::<_, [bool; 4]>(source) { + Ok(value) => { + assert!(valid); + assert_eq!(value, [false, true, false, true]); + } + Err(error) => { + assert!(!valid); + assert!(matches!(error, TryReadError::Validity(_))); + assert!(ptr::eq(error.into_src(), source)); + } + } + + let error = try_read_from::<_, [bool; 5]>(source).unwrap_err(); + assert!(matches!(error, TryReadError::Size(_))); + assert!(ptr::eq(error.into_src(), source)); + } + + for (value, valid) in [(u16::from_ne_bytes([0, 1]), true), (0x0202, false)] { + let source = Source([value; 2]); + check(&source, valid); + let source: &Source<[u16]> = &source; + check(source, valid); + } + + let source = Source([(); 2]); + assert!(try_read_from::<_, ()>(&source).is_ok()); + let source: &Source<[()]> = &source; + assert!(try_read_from::<_, ()>(source).is_ok()); + } + + #[test] + fn test_try_read_initialized_padding() { + #[derive(KnownLayout, Immutable, Debug, PartialEq, Eq)] + #[repr(C, align(8))] + struct Padded(u8); + + // SAFETY: `Padded` has only a `u8` field and padding, so all + // initialized byte sequences are valid. Per + // https://doc.rust-lang.org/1.93.1/reference/behavior-considered-undefined.html#invalid-values: + // + // An integer (`i*`/`u*`), floating point value (`f*`), or raw pointer + // must be initialized, i.e., must not be obtained from uninitialized + // memory. + // + // A `struct`, tuple, and array requires all fields/elements to be + // valid at their respective type. + const _: () = unsafe { + unsafe_impl!(=> TryFromBytes for Padded; |candidate| { + // Observe every byte while validation is running, including + // padding that a typed move of `MaybeUninit` could + // discard. The returned `Padded` need not retain its padding. + assert_eq!(candidate.as_bytes::().as_ref(), &[0xA5; 8]); + true + }) + }; + + let source = [0xA5; 8]; + assert_eq!(Padded::try_read_from_bytes(&source), Ok(Padded(0xA5))); + + let prefix_source = [0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0]; + let (value, suffix) = Padded::try_read_from_prefix(&prefix_source).unwrap(); + assert_eq!(value, Padded(0xA5)); + assert!(ptr::eq(suffix, &prefix_source[8..])); + + let suffix_source = [0, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5]; + let (prefix, value) = Padded::try_read_from_suffix(&suffix_source).unwrap(); + assert_eq!(value, Padded(0xA5)); + assert!(ptr::eq(prefix, &suffix_source[..1])); + } + + #[test] + fn test_try_read_error_sources_and_zero_sized() { + let invalid = [2, 3]; + for (error, source) in [ + (bool::try_read_from_bytes(&invalid[..1]).unwrap_err(), &invalid[..1]), + (bool::try_read_from_prefix(&invalid).unwrap_err(), &invalid[..]), + (bool::try_read_from_suffix(&invalid).unwrap_err(), &invalid[..]), + ] { + assert!(matches!(error, TryReadError::Validity(_))); + assert!(ptr::eq(error.into_src(), source)); + } + for source in [&[][..], &invalid[..]] { + let error = bool::try_read_from_bytes(source).unwrap_err(); + assert!(matches!(error, TryReadError::Size(_))); + assert!(ptr::eq(error.into_src(), source)); + } + for error in [ + bool::try_read_from_prefix(&invalid[..0]).unwrap_err(), + bool::try_read_from_suffix(&invalid[..0]).unwrap_err(), + ] { + assert!(matches!(error, TryReadError::Size(_))); + assert!(ptr::eq(error.into_src(), &invalid[..0])); + } + + assert_eq!(<()>::try_read_from_bytes(&[]), Ok(())); + assert_eq!(<()>::try_read_from_prefix(&invalid), Ok(((), &invalid[..]))); + assert_eq!(<()>::try_read_from_suffix(&invalid), Ok((&invalid[..], ()))); + } + #[test] fn test_ref_from_mut_from_bytes() { // Test `FromBytes::{ref_from_bytes, mut_from_bytes}{,_prefix,Suffix}` @@ -7502,6 +8054,16 @@ mod tests { assert_eq!(&*s, &[0, 3, 0]); } + #[test] + fn test_new_box_zeroed_with_elems_nested_packed_dst() { + let mut value = NestedPackedDstOuter::<[u8]>::new_box_zeroed_with_elems(0).unwrap(); + assert_eq!(mem::size_of_val(&*value), 10); + + // Exercise the whole-object write that exposed the former + // under-allocation under Miri and AddressSanitizer. + value.zero(); + } + #[test] fn test_new_box_zeroed_with_elems_empty() { let s: Box<[u64]> = <[u64]>::new_box_zeroed_with_elems(0).unwrap(); @@ -7571,6 +8133,133 @@ mod tests { assert_eq!(rest.len(), 4); } + #[test] + fn test_ref_from_prefix_suffix_typed_source() { + #[derive(IntoBytes, Immutable)] + #[repr(transparent)] + struct Source(T); + + fn check(source: &S) { + let bytes = source.as_bytes(); + assert_eq!(bytes.len(), 4); + + for cast_type in [CastType::Prefix, CastType::Suffix] { + let (expected, expected_rest) = match cast_type { + CastType::Prefix => (&bytes[..3], &bytes[3..]), + CastType::Suffix => (&bytes[1..], &bytes[..1]), + }; + for meta in [None, Some(1)] { + let (value, rest) = + ref_from_prefix_suffix::<_, [[u8; 3]]>(source, meta, cast_type).unwrap(); + assert_eq!(value, &[expected]); + assert!(ptr::eq(value.as_bytes(), expected)); + assert!(ptr::eq(rest, expected_rest)); + } + + let err = + ref_from_prefix_suffix::<_, [u8; 5]>(source, None, cast_type).unwrap_err(); + assert!(matches!(err, CastError::Size(_))); + assert!(ptr::eq(err.into_src(), source)); + + let err = + ref_from_prefix_suffix::<_, [u8]>(source, Some(5), cast_type).unwrap_err(); + assert!(matches!(err, CastError::Size(_))); + assert!(ptr::eq(err.into_src(), source)); + } + } + + check(&Source([0x0102u16, 0x0304])); + let source = Source([false, true, false, true]); + let source: &Source<[bool]> = &source; + check(source); + } + + #[test] + fn test_ref_from_prefix_suffix_typed_source_alignment_error() { + let storage = Align::<[bool; 9], AU64>::new([false; 9]); + for cast_type in [CastType::Prefix, CastType::Suffix] { + let source = match cast_type { + CastType::Prefix => &storage.t[1..], + CastType::Suffix => &storage.t[..], + }; + let err = ref_from_prefix_suffix::<_, AU64>(source, None, cast_type).unwrap_err(); + assert!(matches!(err, CastError::Alignment(_))); + let recovered: &[bool] = err.into_src(); + assert!(ptr::eq(recovered, source)); + + let err = try_ref_from_prefix_suffix::<_, AU64>(source, cast_type, None).unwrap_err(); + assert!(matches!(err, TryCastError::Alignment(_))); + let recovered: &[bool] = err.into_src(); + assert!(ptr::eq(recovered, source)); + } + } + + #[test] + fn test_try_ref_from_prefix_suffix_typed_source() { + #[derive(IntoBytes, Immutable)] + #[repr(transparent)] + struct Source(T); + + fn check(source: &S) { + let bytes = source.as_bytes(); + assert_eq!(bytes, [0, 1, 0, 1]); + + for cast_type in [CastType::Prefix, CastType::Suffix] { + let (expected, expected_rest) = match cast_type { + CastType::Prefix => (&bytes[..3], &bytes[3..]), + CastType::Suffix => (&bytes[1..], &bytes[..1]), + }; + for meta in [None, Some(1)] { + let (value, rest) = + try_ref_from_prefix_suffix::<_, [[bool; 3]]>(source, cast_type, meta) + .unwrap(); + assert_eq!(value.as_bytes(), expected); + assert!(ptr::eq(value.as_bytes(), expected)); + assert!(ptr::eq(rest, expected_rest)); + } + + let err = try_ref_from_prefix_suffix::<_, [bool; 5]>(source, cast_type, None) + .unwrap_err(); + assert!(matches!(err, TryCastError::Size(_))); + assert!(ptr::eq(err.into_src(), source)); + + let err = try_ref_from_prefix_suffix::<_, [bool]>(source, cast_type, Some(5)) + .unwrap_err(); + assert!(matches!(err, TryCastError::Size(_))); + assert!(ptr::eq(err.into_src(), source)); + } + } + + check(&Source([u16::from_ne_bytes([0, 1]); 2])); + let source = Source([false, true, false, true]); + let source: &Source<[bool]> = &source; + check(source); + } + + #[test] + fn test_try_ref_from_prefix_suffix_typed_source_validity_error() { + fn check(source: &S) { + for cast_type in [CastType::Prefix, CastType::Suffix] { + let err = + try_ref_from_prefix_suffix::<_, bool>(source, cast_type, None).unwrap_err(); + assert!(matches!(err, TryCastError::Validity(_))); + assert!(ptr::eq(err.into_src(), source)); + + for meta in [None, Some(1)] { + let err = try_ref_from_prefix_suffix::<_, [bool]>(source, cast_type, meta) + .unwrap_err(); + assert!(matches!(err, TryCastError::Validity(_))); + assert!(ptr::eq(err.into_src(), source)); + } + } + } + + let source = [u16::from_ne_bytes([2, 2]); 2]; + check(&source); + check(&source[..]); + check(source.as_bytes()); + } + #[test] fn test_try_ref_from_prefix_suffix() { use crate::util::testutil::Align; diff --git a/rust/zerocopy/src/macros.rs b/rust/zerocopy/src/macros.rs index ec67c03a44fc4d..73f6dd0ca0ae0a 100644 --- a/rust/zerocopy/src/macros.rs +++ b/rust/zerocopy/src/macros.rs @@ -1033,7 +1033,7 @@ macro_rules! cryptocorrosion_derive_traits { )? { #[inline(always)] - fn is_bit_valid(_: $crate::Maybe<'_, Self, A>) -> bool + fn is_safe(_: $crate::Maybe<'_, Self, A>) -> bool where A: $crate::invariant::Alignment, { @@ -1179,7 +1179,7 @@ macro_rules! cryptocorrosion_derive_traits { )* { #[inline(always)] - fn is_bit_valid(_: $crate::Maybe<'_, Self, A>) -> bool + fn is_safe(_: $crate::Maybe<'_, Self, A>) -> bool where A: $crate::invariant::Alignment, { @@ -1411,6 +1411,38 @@ mod tests { let x: &SliceDst = transmute_ref!(slice_dst_big); assert_eq!(x, slice_dst_small); + // A packed outer DST can preserve the trailing field's stronger + // rounding alignment. These two types consequently have different + // normalized layout formulas even though both have size `2 + 4 * n` + // for every trailing slice length. Ensure the semantic size-sequence + // comparison accepts this representation-preserving transmutation. + #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)] + #[repr(C, align(2))] + struct TransmuteAlign2([u8; 4]); + + #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)] + #[repr(C, packed(2))] + struct TransmutePacked4 { + head: u16, + tail: [u32], + } + + #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)] + #[repr(C, packed(2))] + struct TransmutePacked2 { + head: u16, + tail: [TransmuteAlign2], + } + + #[repr(C, align(2))] + struct TransmuteBytes([u8; 10]); + + let bytes = TransmuteBytes([0; 10]); + let src = TransmutePacked4::ref_from_bytes(&bytes.0).unwrap(); + let dst: &TransmutePacked2 = transmute_ref!(src); + assert_eq!(mem::size_of_val(src), 10); + assert_eq!(mem::size_of_val(dst), 10); + // Test that it's legal to transmute a reference while shrinking the // lifetime (note that `X` has the lifetime `'static`). let x: &[u8; 8] = transmute_ref!(X); @@ -1471,6 +1503,19 @@ mod tests { // its `PanicOnDrop` temporary, which would cause a panic. assert_eq!(y.as_ref().map_err(|p| &p.src.0), Err::<&bool, _>(&2u8)); mem::forget(y); + + #[derive(Debug, TryFromBytes)] + #[repr(C, align(8))] + struct PaddedBool(bool); + + // A failed transmutation must return every source byte, including + // bytes which would be padding in the destination type. + let src = [2u8, 1, 2, 3, 4, 5, 6, 7]; + let x: Result = try_transmute!(src); + assert_eq!(x.unwrap_err().into_src(), src); + + let x: Result = try_transmute!([0u8; 8]); + assert!(!x.unwrap().0); } #[test] @@ -1687,9 +1732,9 @@ mod tests { #[test] fn test_include_value() { const AS_U32: u32 = include_value!("../testdata/include_value/data"); - assert_eq!(AS_U32, u32::from_ne_bytes([b'a', b'b', b'c', b'd'])); + assert_eq!(AS_U32, u32::from_ne_bytes(*b"abcd")); const AS_I32: i32 = include_value!("../testdata/include_value/data"); - assert_eq!(AS_I32, i32::from_ne_bytes([b'a', b'b', b'c', b'd'])); + assert_eq!(AS_I32, i32::from_ne_bytes(*b"abcd")); } #[test] diff --git a/rust/zerocopy/src/pointer/inner.rs b/rust/zerocopy/src/pointer/inner.rs index 949b60a3f83eae..8beb6267b2176c 100644 --- a/rust/zerocopy/src/pointer/inner.rs +++ b/rust/zerocopy/src/pointer/inner.rs @@ -289,10 +289,10 @@ where /// /// Given `let (left, right) = ptr.split_at(l_len)`, it is guaranteed that /// `left` and `right` are contiguous and non-overlapping if - /// `l_len.padding_needed_for() == 0`. This is true for all `[T]`. + /// the left part has no trailing padding. This is true for all `[T]`. /// - /// If `l_len.padding_needed_for() != 0`, then the left pointer will overlap - /// the right pointer to satisfy `T`'s padding requirements. + /// Any trailing padding in the left part begins at the right part's + /// starting address and may overlap its referent. #[inline] #[must_use] pub unsafe fn split_at_unchecked( @@ -313,23 +313,19 @@ where // Trivially, `slf.meta() <= slf.meta()`. let right = unsafe { right.slice_unchecked(l_len..self.meta().get()) }; - // SAFETY: If `l_len.padding_needed_for() == 0`, then `left` and `right` - // are non-overlapping. Proof: `left` is constructed `slf` with `l_len` - // as its (exclusive) upper bound. If `l_len.padding_needed_for() == 0`, - // then `left` requires no trailing padding following its final element. - // Since `right` is constructed from `slf`'s trailing slice with `l_len` - // as its (inclusive) lower bound, no byte is referred to by both - // pointers. + // SAFETY: If the left part has no trailing padding, then `left` and + // `right` are non-overlapping. Proof: `left` has `l_len` trailing + // elements, and `right` starts at element `l_len` of the original + // trailing slice. Without padding after `left`'s final element, no + // byte is referred to by both pointers. // - // Conversely, `l_len.padding_needed_for() == N`, where `N - // > 0`, `left` requires `N` bytes of trailing padding following its - // final element. Since `right` is constructed from the trailing slice - // of `slf` with `l_len` as its (inclusive) lower bound, the first `N` - // bytes of `right` are aliased by `left`. + // Any trailing padding in `left` begins at `right`'s starting + // address. If `right` is nonzero sized, the referents overlap. (left, right) } - /// Produces the trailing slice of `self`. + /// Produces the trailing slice of `self` using `T::Elem` as its element + /// representation. #[inline] #[must_use] pub fn trailing_slice(self) -> PtrInner<'a, [T::Elem]> @@ -368,9 +364,10 @@ where // 1. If `ptr`'s referent is not zero sized, then `ptr` has valid // provenance for `A` because `raw` is derived from the same // allocated object as `self` via provenance-preserving operations. - // 2. If `ptr`'s referent is not zero sized, then `ptr` addresses a byte - // range which is entirely contained in `A`, by previous safety proof - // on `bytes`. + // 2. `T: SplitAt` guarantees that `T::Elem` has the size of the actual + // trailing element. The new slice has the same length and start as + // the original trailing slice, so it covers the same byte range. + // If nonzero sized, that range is entirely contained in `A`. // 3. `ptr` addresses a byte range whose length fits in an `isize`, by // consequence of #2. // 4. `ptr` addresses a byte range which does not wrap around the diff --git a/rust/zerocopy/src/pointer/invariant.rs b/rust/zerocopy/src/pointer/invariant.rs index 7ff0d43dad5e89..8dfaf82e134f4e 100644 --- a/rust/zerocopy/src/pointer/invariant.rs +++ b/rust/zerocopy/src/pointer/invariant.rs @@ -47,7 +47,7 @@ pub trait Alignment: Sealed { fn read(ptr: crate::Ptr<'_, T, I>) -> T where T: Copy + Read, - I: Invariants, + I: Invariants, I::Aliasing: Reference; } @@ -64,35 +64,39 @@ pub trait Alignment: Sealed { /// all `T: ?Sized`. Any `V: Validity` must guarantee that this set is only a /// function of the *bit validity* of the referent type, `T`, and not of any /// other property of `T`. As a consequence, given `V: Validity`, `T`, and `U` -/// where `T` and `U` have the same bit validity, `S(V, T) = S(V, U)`. +/// where `T` and `U` have the same bit validity, `S(T, V) = S(U, V)`. /// /// It is guaranteed that the referent of any `ptr: Ptr` is a member of /// `S(T, V)`. Unsafe code must ensure that this guarantee will be upheld for /// any existing `Ptr`s or any `Ptr`s that that code creates. /// -/// An important implication of this guarantee is that it restricts what -/// transmutes are sound, where "transmute" is used in this context to refer to -/// changing the referent type or validity invariant of a `Ptr`, as either -/// change may change the set of bit values permitted to appear in the referent. -/// In particular, the following are necessary (but not sufficient) conditions -/// in order for a transmute from `src: Ptr` to `dst: Ptr` to be -/// sound: -/// - If `S(T, V) = S(U, W)`, then no restrictions apply; otherwise, +/// An important implication of this guarantee is that it restricts which exact +/// reinterpretations are sound. Here, an exact reinterpretation changes the +/// referent type or validity invariant of a `Ptr` while preserving the exact +/// byte range. In particular, given `src: Ptr` and `dst: Ptr` which +/// refer to the same byte range, the following are necessary (but not +/// sufficient) conditions: +/// - If `S(T, V) = S(U, W)`, then no additional validity-preservation +/// restrictions apply; otherwise, /// - If `dst` permits mutation of its referent (e.g. via `Exclusive` aliasing /// or interior mutation under `Shared` aliasing), then it must hold that -/// `S(T, V) ⊇ S(U, W)` - in other words, the transmute must not expand the -/// set of allowed referent bit patterns. A violation of this requirement +/// `S(T, V) ⊇ S(U, W)` - in other words, the reinterpretation must not expand +/// the set of allowed referent bit patterns. A violation of this requirement /// would permit using `dst` to write `x` where `x ∈ S(U, W)` but `x ∉ S(T, /// V)`, which would violate the guarantee that `src`'s referent may only /// contain values in `S(T, V)`. /// - If the referent may be mutated without going through `dst` while `dst` is /// live (e.g. via interior mutation on a `Shared`-aliased `Ptr` or `&` /// reference), then it must hold that `S(T, V) ⊆ S(U, W)` - in other words, -/// the transmute must not shrink the set of allowed referent bit patterns. A -/// violation of this requirement would permit using `src` or another -/// mechanism (e.g. a `&` reference used to derive `src`) to write `x` where -/// `x ∈ S(T, V)` but `x ∉ S(U, W)`, which would violate the guarantee that -/// `dst`'s referent may only contain values in `S(U, W)`. +/// the reinterpretation must not shrink the set of allowed referent bit +/// patterns. A violation of this requirement would permit using `src` or +/// another mechanism (e.g. a `&` reference used to derive `src`) to write `x` +/// where `x ∈ S(T, V)` but `x ∉ S(U, W)`, which would violate the guarantee +/// that `dst`'s referent may only contain values in `S(U, W)`. +/// +/// These conditions describe only exact reinterpretations. Shrinking +/// projections may have validity that depends on the relationship between the +/// projected region and bytes outside it, and require additional reasoning. pub unsafe trait Validity: Sealed { const KIND: ValidityKind; } @@ -101,7 +105,7 @@ pub enum ValidityKind { Uninit, AsInitialized, Initialized, - Valid, + Safe, } /// An [`Aliasing`] invariant which is either [`Shared`] or [`Exclusive`]. @@ -147,7 +151,7 @@ impl Alignment for Unaligned { fn read(ptr: crate::Ptr<'_, T, I>) -> T where T: Copy + Read, - I: Invariants, + I: Invariants, I::Aliasing: Reference, { (*ptr.into_unalign().as_ref()).into_inner() @@ -162,7 +166,7 @@ impl Alignment for Aligned { fn read(ptr: crate::Ptr<'_, T, I>) -> T where T: Copy + Read, - I: Invariants, + I: Invariants, I::Aliasing: Reference, { *ptr.as_ref() @@ -227,13 +231,20 @@ unsafe impl Validity for Initialized { /// The referent of a `Ptr` is valid for `T`, upholding bit validity and any /// library safety invariants. -pub enum Valid {} -// SAFETY: `Valid`'s validity is well-defined for all `T: ?Sized`, and is not a +pub enum Safe {} +// SAFETY: `Safe`'s validity is well-defined for all `T: ?Sized`, and is not a // function of any property of `T` other than its bit validity. -unsafe impl Validity for Valid { - const KIND: ValidityKind = ValidityKind::Valid; +unsafe impl Validity for Safe { + const KIND: ValidityKind = ValidityKind::Safe; } +/// Proof helper for casts whose validity requirement does not depend on the +/// referent type. +/// +/// Currently this covers `Uninit`, which permits any byte state, and +/// `Initialized`, which requires every referent byte to be initialized +/// regardless of referent type. +/// /// # Safety /// /// `DT: CastableFrom` is sound if `SV = DV = Uninit` or `SV = DV = @@ -289,7 +300,7 @@ mod sealed { impl Sealed for Uninit {} impl Sealed for AsInitialized {} impl Sealed for Initialized {} - impl Sealed for Valid {} + impl Sealed for Safe {} impl Sealed for (A, AA, V) {} diff --git a/rust/zerocopy/src/pointer/mod.rs b/rust/zerocopy/src/pointer/mod.rs index d6eacc52febe2e..0ce54339153099 100644 --- a/rust/zerocopy/src/pointer/mod.rs +++ b/rust/zerocopy/src/pointer/mod.rs @@ -25,9 +25,9 @@ pub use transmute::*; use crate::wrappers::ReadOnly; /// A shorthand for a maybe-valid, maybe-aligned reference. Used as the argument -/// to [`TryFromBytes::is_bit_valid`]. +/// to [`TryFromBytes::is_safe`]. /// -/// [`TryFromBytes::is_bit_valid`]: crate::TryFromBytes::is_bit_valid +/// [`TryFromBytes::is_safe`]: crate::TryFromBytes::is_safe pub type Maybe<'a, T, Alignment = invariant::Unaligned> = Ptr<'a, ReadOnly, (invariant::Shared, Alignment, invariant::Initialized)>; @@ -47,10 +47,7 @@ where pub mod cast { use core::{marker::PhantomData, mem}; - use crate::{ - layout::{SizeInfo, TrailingSliceLayout}, - HasField, KnownLayout, PtrInner, - }; + use crate::{layout::SizeInfo, HasField, KnownLayout, PtrInner}; /// A pointer cast or projection. /// @@ -182,11 +179,9 @@ pub mod cast { // SAFETY: By the `static_assert!`, `Src` and `Dst` are either: // - Both sized and equal in size - // - Both slice DSTs with the same trailing slice offset and element size - // and with align_of::() == align_of::(). These ensure that any - // given pointer metadata encodes the same size for both `Src` and `Dst` - // (note that the alignment is required as it affects the amount of - // trailing padding). Thus, `project` preserves the set of referent bytes. + // - Both slice DSTs with the same trailing-slice offset, the same object + // size for every metadata value, and with align_of::() == + // align_of::(). Thus, `project` preserves the set of referent bytes. unsafe impl Project for CastUnsized where Src: ?Sized + KnownLayout, @@ -201,10 +196,11 @@ pub mod cast { let dst = ::LAYOUT; match (src.size_info, dst.size_info) { (SizeInfo::Sized { size: src_size }, SizeInfo::Sized { size: dst_size }) => src_size == dst_size, - ( - SizeInfo::SliceDst(TrailingSliceLayout { offset: src_offset, elem_size: src_elem_size }), - SizeInfo::SliceDst(TrailingSliceLayout { offset: dst_offset, elem_size: dst_elem_size }) - ) => src.align.get() == dst.align.get() && src_offset == dst_offset && src_elem_size == dst_elem_size, + (SizeInfo::SliceDst(src_trailing), SizeInfo::SliceDst(dst_trailing)) => { + src.align.get() == dst.align.get() + && src_trailing.offset == dst_trailing.offset + && src_trailing.has_same_size_sequence(dst_trailing) + }, _ => false, } }); @@ -225,10 +221,8 @@ pub mod cast { // SAFETY: By the `static_assert!` in `Project::project`, `Src` and `Dst` // are either: // - Both sized and equal in size - // - Both slice DSTs with the same alignment, trailing slice offset, and - // element size. These ensure that any given pointer metadata encodes the - // same size for both `Src` and `Dst` (note that the alignment is required - // as it affects the amount of trailing padding). + // - Both slice DSTs with the same alignment and trailing-slice offset, and + // with equal sizes for every metadata value. unsafe impl CastExact for CastUnsized where Src: ?Sized + KnownLayout, @@ -240,18 +234,22 @@ pub mod cast { /// /// A `Projection` is a [`Project`] which implements projection by /// delegating to an implementation of [`HasField::project`]. + /// + /// The `Client` parameter exists to select between implementations of + /// [`HasField`] that would otherwise conflict. #[allow(missing_debug_implementations, missing_copy_implementations)] - pub struct Projection { + pub struct Projection { _never: core::convert::Infallible, + _client: PhantomData, _phantom: PhantomData, } // SAFETY: `HasField::project` has the same safety post-conditions as // `Project::project`. - unsafe impl Project - for Projection + unsafe impl + Project for Projection where - T: HasField, + T: HasField, { #[inline(always)] fn project(src: PtrInner<'_, T>) -> *mut T::Type { @@ -292,10 +290,10 @@ pub mod cast { // // FIXME(https://github.com/rust-lang/unsafe-code-guidelines/issues/595): // Cite the documentation once it's updated. - unsafe impl Cast - for Projection + unsafe impl Cast + for Projection where - T: HasField, + T: HasField, { } diff --git a/rust/zerocopy/src/pointer/ptr.rs b/rust/zerocopy/src/pointer/ptr.rs index 7213f6f4a04ee8..0fcd2ed11d372e 100644 --- a/rust/zerocopy/src/pointer/ptr.rs +++ b/rust/zerocopy/src/pointer/ptr.rs @@ -169,7 +169,7 @@ mod _conversions { use crate::pointer::cast::{CastExact, CastSized, IdCast}; /// `&'a T` → `Ptr<'a, T>` - impl<'a, T> Ptr<'a, T, (Shared, Aligned, Valid)> + impl<'a, T> Ptr<'a, T, (Shared, Aligned, Safe)> where T: 'a + ?Sized, { @@ -183,20 +183,19 @@ mod _conversions { // 1. `ptr`, by invariant on `&'a T`, conforms to the alignment // invariant of `Aligned`. // 2. `ptr`'s referent, by invariant on `&'a T`, is a bit-valid `T`. - // This satisfies the requirement that a `Ptr` + // This satisfies the requirement that a `Ptr` // point to a bit-valid `T`. Even if `T` permits interior - // mutation, this invariant guarantees that the returned `Ptr` - // can only ever be used to modify the referent to store - // bit-valid `T`s, which ensures that the returned `Ptr` cannot - // be used to violate the soundness of the original `ptr: &'a T` - // or of any other references that may exist to the same - // referent. + // mutation, this invariant guarantees that the returned `Ptr` can + // only ever be used to modify the referent to store bit-valid + // `T`s, which ensures that the returned `Ptr` cannot be used to + // violate the soundness of the original `ptr: &'a T` or of any + // other references that may exist to the same referent. unsafe { Self::from_inner(inner) } } } /// `&'a mut T` → `Ptr<'a, T>` - impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)> + impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Safe)> where T: 'a + ?Sized, { @@ -211,7 +210,7 @@ mod _conversions { // invariant of `Aligned`. // 2. `ptr`'s referent, by invariant on `&'a mut T`, is a bit-valid // `T`. This satisfies the requirement that a `Ptr` point to a bit-valid `T`. This invariant guarantees + // Safe)>` point to a bit-valid `T`. This invariant guarantees // that the returned `Ptr` can only ever be used to modify the // referent to store bit-valid `T`s, which ensures that the // returned `Ptr` cannot be used to violate the soundness of the @@ -224,7 +223,7 @@ mod _conversions { impl<'a, T, I> Ptr<'a, T, I> where T: 'a + ?Sized, - I: Invariants, + I: Invariants, I::Aliasing: Reference, { /// Converts `self` to a shared reference. @@ -239,7 +238,7 @@ mod _conversions { let raw = self.as_inner().as_non_null(); // SAFETY: `self` satisfies the `Aligned` invariant, so we know that // `raw` is validly-aligned for `T`. - #[cfg(miri)] + #[cfg(all(miri, __ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE))] unsafe { crate::util::miri_promise_symbolic_alignment( raw.as_ptr().cast(), @@ -264,7 +263,7 @@ mod _conversions { // // 3. The pointer must point to a validly-initialized instance of // `T`. This is ensured by-contract on `Ptr`, because the - // `I::Validity` is `Valid`. + // `I::Validity` is `Safe`. // // 4. You must enforce Rust’s aliasing rules. This is ensured by // contract on `Ptr`, because `I::Aliasing: Reference`. Either it @@ -377,7 +376,7 @@ mod _conversions { } /// `Ptr<'a, T>` → `&'a mut T` - impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)> + impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Safe)> where T: 'a + ?Sized, { @@ -389,7 +388,7 @@ mod _conversions { let mut raw = self.as_inner().as_non_null(); // SAFETY: `self` satisfies the `Aligned` invariant, so we know that // `raw` is validly-aligned for `T`. - #[cfg(miri)] + #[cfg(all(miri, __ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE))] unsafe { crate::util::miri_promise_symbolic_alignment( raw.as_ptr().cast(), @@ -413,8 +412,8 @@ mod _conversions { // This is ensured by contract on all `PtrInner`s. // // 3. The pointer must point to a validly-initialized instance of - // `T`. This is ensured by-contract on `Ptr`, because the - // validity invariant is `Valid`. + // `T`. This is ensured by-contract on `Ptr`, because the validity + // invariant is `Safe`. // // 4. You must enforce Rust’s aliasing rules. This is ensured by // contract on `Ptr`, because the `ALIASING_INVARIANT` is @@ -431,6 +430,10 @@ mod _conversions { where I: Invariants, { + /// Reinterprets the same byte region as `U` with validity `V`. + /// + /// This preserves the aliasing invariant, uses [`SizeEq`] to select a + /// [`CastExact`] implementation, and conservatively forgets alignment. #[must_use] #[inline(always)] pub fn transmute(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)> @@ -443,6 +446,12 @@ mod _conversions { self.transmute_with::>::CastFrom, R>() } + /// Reinterprets the same byte region as `U` with validity `V` using + /// `C`. + /// + /// `C: CastExact` preserves the byte region. The aliasing invariant is + /// preserved, while alignment is conservatively forgotten because the + /// destination type may have a different alignment requirement. #[inline] #[must_use] pub fn transmute_with(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)> @@ -462,11 +471,15 @@ mod _conversions { // at the same time, as neither can perform interior mutation // - It is directly guaranteed that it is sound for shared code to // operate on these references simultaneously - // - By `U: TransmuteFromPtr`, it + // - By `U: TransmuteFromPtr`, it // is sound to perform this transmute using `C`. unsafe { self.project_transmute_unchecked::<_, _, C>() } } + /// Changes only the validity invariant to `V`. + /// + /// The referent type and byte region are unchanged, so this preserves + /// the existing aliasing and alignment invariants. #[inline] #[must_use] pub fn recall_validity(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, V)> @@ -562,7 +575,7 @@ mod _conversions { // FIXME(#1359): This should be a `transmute_with` call. // Unfortunately, to avoid blanket impl conflicts, we only implement // `TransmuteFrom` for `Unalign` (and vice versa) specifically - // for `Valid` validity, not for all validity types. + // for `Safe` validity, not for all validity types. // SAFETY: // - By `CastSized: Cast`, `CastSized` preserves referent address, @@ -589,7 +602,7 @@ mod _conversions { impl<'a, T, I> Ptr<'a, T, I> where T: ?Sized, - I: Invariants, + I: Invariants, I::Aliasing: Reference, { /// Reads the referent. @@ -656,6 +669,22 @@ mod _transitions { unsafe { self.assume_invariants::() } } + /// Downgrades `self` to shared aliasing. + #[inline] + #[must_use] + pub(super) fn into_shared(self) -> Ptr<'a, T, (Shared, I::Alignment, I::Validity)> + where + I::Aliasing: Reference, + { + // SAFETY: `I::Aliasing: Reference` guarantees that the source + // aliasing is either `Shared` or `Exclusive`. `Shared` requires no + // transition. If it is `Exclusive`, consuming `self` ends the only + // permitted access to the referent, so the returned pointer can + // soundly carry `Shared` aliasing. The referent type, alignment, + // and validity invariants are unchanged. + unsafe { self.assume_invariants() } + } + /// Assumes that `self`'s referent is validly-aligned for `T` if /// required by `A`. /// @@ -738,40 +767,40 @@ mod _transitions { unsafe { self.assume_validity::() } } - /// A shorthand for `self.assume_validity()`. + /// A shorthand for `self.assume_validity()`. /// /// # Safety /// /// The caller promises to uphold the safety preconditions of - /// `self.assume_validity()`. + /// `self.assume_validity()`. #[must_use] #[inline] - pub unsafe fn assume_valid(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Valid)> { + pub unsafe fn assume_safe(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Safe)> { // SAFETY: The caller has promised to uphold the safety // preconditions. - unsafe { self.assume_validity::() } + unsafe { self.assume_validity::() } } /// Checks that `self`'s referent is validly initialized for `T`, - /// returning a `Ptr` with `Valid` on success. + /// returning a `Ptr` with `Safe` on success. /// /// # Panics /// /// This method will panic if - /// [`T::is_bit_valid`][TryFromBytes::is_bit_valid] panics. + /// [`T::is_safe`][TryFromBytes::is_safe] panics. /// /// # Safety /// /// On error, unsafe code may rely on this method's returned /// `ValidityError` containing `self`. #[inline] - pub fn try_into_valid( + pub fn try_into_safe( mut self, - ) -> Result, ValidityError> + ) -> Result, ValidityError> where T: TryFromBytes + Read - + TryTransmuteFromPtr, + + TryTransmuteFromPtr, ReadOnly: Read, I::Aliasing: Reference, I: Invariants, @@ -779,14 +808,13 @@ mod _transitions { // This call may panic. If that happens, it doesn't cause any // soundness issues, as we have not generated any invalid state // which we need to fix before returning. - if T::is_bit_valid(self.reborrow().transmute::<_, _, _>().reborrow_shared()) { - // SAFETY: If `T::is_bit_valid`, code may assume that `self` - // contains a bit-valid instance of `T`. By `T: - // TryTransmuteFromPtr`, so - // long as `self`'s referent conforms to the `Valid` validity - // for `T` (which we just confirmed), then this transmute is - // sound. - Ok(unsafe { self.assume_valid() }) + if T::is_safe(self.reborrow().transmute::<_, _, _>().reborrow_shared()) { + // SAFETY: If `T::is_safe` returns true, code may assume that + // `self` contains a valid `T`, so its referent conforms to + // `Safe` for `T`. By `T: TryTransmuteFromPtr`, given that condition, changing `self`'s + // validity to `Safe` is sound. + Ok(unsafe { self.assume_safe() }) } else { Err(ValidityError::new(self)) } @@ -803,7 +831,7 @@ mod _transitions { } /// Casts of the referent type. -#[cfg_attr(not(zerocopy_unstable_ptr), allow(unreachable_pub))] +#[allow(unreachable_pub)] // False positive on MSRV pub use _casts::TryWithError; mod _casts { use core::cell::UnsafeCell; @@ -875,19 +903,22 @@ mod _casts { } #[inline(always)] - pub fn project( + pub fn project( mut self, ) -> Result, T::Error> where - T: ProjectField, + T: ProjectField, I::Aliasing: Reference, { use crate::pointer::cast::Projection; - match T::is_projectable(self.reborrow().project_tag()) { + match >::is_projectable( + self.reborrow().project_tag::(), + ) { Ok(()) => { let inner = self.as_inner(); - let projected = inner.project::<_, Projection>(); - // SAFETY: By `T: ProjectField`, + let projected = + inner.project::<_, Projection>(); + // SAFETY: By `T: ProjectField`, // for `self: Ptr<'_, T, I>` such that `T::is_projectable` // (which we've verified in this match arm), // `T::project(self.as_inner())` conforms to @@ -908,17 +939,22 @@ mod _casts { #[must_use] #[inline(always)] - pub(crate) fn project_tag(self) -> Ptr<'a, T::Tag, I> + pub fn project_tag( + self, + ) -> Ptr<'a, >::Tag, (Shared, I::Alignment, I::Validity)> where - T: HasTag, + T: HasTag, + I::Aliasing: Reference, { - // SAFETY: By invariant on `Self::ProjectToTag`, this is a sound - // projection. - let tag = unsafe { self.project_transmute_unchecked::<_, _, T::ProjectToTag>() }; - // SAFETY: By invariant on `Self::ProjectToTag`, the projected - // pointer has the same alignment as `ptr`. - let tag = unsafe { tag.assume_alignment() }; - tag.unify_invariants() + let ptr = self.into_shared(); + // SAFETY: By invariant on `ProjectToTag`, it is sound to project + // `ptr` to a shared tag pointer with the same validity invariant. + let ptr = unsafe { + ptr.project_transmute_unchecked::<_, _, >::ProjectToTag>() + }; + // SAFETY: By invariant on `ProjectToTag`, the projected pointer has + // the same alignment as `self`. + unsafe { ptr.assume_alignment() } } /// Attempts to transform the pointer, restoring the original on @@ -1015,11 +1051,11 @@ mod _casts { #[allow(clippy::wrong_self_convention)] #[must_use] #[inline] - pub fn as_bytes(self) -> Ptr<'a, [u8], (I::Aliasing, Aligned, Valid)> + pub fn as_bytes(self) -> Ptr<'a, [u8], (I::Aliasing, Aligned, Safe)> where - [u8]: TransmuteFromPtr, + [u8]: TransmuteFromPtr, { - self.transmute_with::<[u8], Valid, AsBytesCast, _>().bikeshed_recall_aligned() + self.transmute_with::<[u8], Safe, AsBytesCast, _>().bikeshed_recall_aligned() } } @@ -1071,7 +1107,7 @@ mod _casts { /// alignment of `[u8]` is 1. impl<'a, I> Ptr<'a, [u8], I> where - I: Invariants, + I: Invariants, { /// Attempts to cast `self` to a `U` using the given cast type. /// @@ -1135,7 +1171,7 @@ mod _casts { // it is derived from `try_cast_into`, which promises that the // object described by `target` is validly aligned for `U`. // 2. By trait bound, `self` - and thus `target` - is a bit-valid - // `[u8]`. `Ptr<[u8], (_, _, Valid)>` and `Ptr<_, (_, _, + // `[u8]`. `Ptr<[u8], (_, _, Safe)>` and `Ptr<_, (_, _, // Initialized)>` have the same bit validity, and so neither // `self` nor `res` can be used to write a value to the referent // which violates the other's validity invariant. @@ -1143,14 +1179,14 @@ mod _casts { // SAFETY: // 0. `self` and `remainder` both have the type `[u8]`. Thus, they - // have `UnsafeCell`s at the same locations. Type casting does - // not affect aliasing. + // have `UnsafeCell`s at the same locations. Type casting does not + // affect aliasing. // 1. `[u8]` has no alignment requirement. - // 2. `self` has validity `Valid` and has type `[u8]`. Since + // 2. `self` has validity `Safe` and has type `[u8]`. Since // `remainder` references a subset of `self`'s referent, it is // also a bit-valid `[u8]`. Thus, neither `self` nor `remainder` - // can be used to write a value to the referent which violates - // the other's validity invariant. + // can be used to write a value to the referent which violates the + // other's validity invariant. let remainder = unsafe { Ptr::from_inner(remainder) }; Ok((res, remainder)) @@ -1333,6 +1369,23 @@ mod tests { use crate::util::AsAddress; use crate::{pointer::BecauseImmutable, util::testutil::AU64, FromBytes, Immutable}; + #[test] + fn test_project_tag_downgrades_aliasing() { + #[allow(dead_code)] + #[derive(zerocopy_derive::Project)] + #[repr(u8)] + enum Enum { + Variant(u8), + } + + let mut value = Enum::Variant(0); + let _: Ptr< + '_, + >::Tag, + (Shared, Aligned, Safe), + > = Ptr::from_mut(&mut value).project_tag::(); + } + mod test_ptr_try_cast_into_soundness { use super::*; diff --git a/rust/zerocopy/src/pointer/transmute.rs b/rust/zerocopy/src/pointer/transmute.rs index ef9836698203a2..75f9312f622e23 100644 --- a/rust/zerocopy/src/pointer/transmute.rs +++ b/rust/zerocopy/src/pointer/transmute.rs @@ -24,14 +24,14 @@ use crate::{ FromBytes, Immutable, IntoBytes, Unalign, }; -/// Transmutations which are sound to attempt, conditional on validating the bit -/// validity of the destination type. +/// Exact pointer reinterpretations which are sound to attempt, conditional on +/// establishing the destination validity invariant. /// /// If a `Ptr` transmutation is `TryTransmuteFromPtr`, then it is sound to /// perform that transmutation so long as some additional mechanism is used to -/// validate that the referent is bit-valid for the destination type. That -/// validation mechanism could be a type bound (such as `TransmuteFrom`) or a -/// runtime validity check. +/// establish that the referent is `DV`-valid for the destination type. That +/// mechanism could be a type bound (such as `TransmuteFrom`) or a runtime +/// validity check. /// /// # Safety /// @@ -49,14 +49,14 @@ use crate::{ /// Given `src: Ptr` and `dst: Ptr`, /// `Dst: TryTransmuteFromPtr` is sound if all of the /// following hold: -/// - Forwards transmutation: Either of the following hold: +/// - Preserve destination validity: Either of the following hold: /// - So long as `dst` is active, no mutation of `dst`'s referent is allowed /// except via `dst` itself /// - The set of `DV`-valid referents of `dst` is a superset of the set of /// `SV`-valid referents of `src` (NOTE: this condition effectively bans /// shrinking or overwriting transmutes, which cannot satisfy this /// condition) -/// - Reverse transmutation: Either of the following hold: +/// - Preserve source validity: Either of the following hold: /// - `dst` does not permit mutation of its referent /// - The set of `DV`-valid referents of `dst` is a subset of the set of /// `SV`-valid referents of `src` (NOTE: this condition effectively bans @@ -126,7 +126,7 @@ pub unsafe trait TryTransmuteFromPtr< pub enum BecauseMutationCompatible {} // SAFETY: -// - Forwards transmutation: By `Dst: MutationCompatible`, we +// - Preserve destination validity: By `Dst: MutationCompatible`, we // know that at least one of the following holds: // - So long as `dst: Ptr` is active, no mutation of its referent is // allowed except via `dst` itself if either of the following hold: @@ -138,17 +138,18 @@ pub enum BecauseMutationCompatible {} // referent as `src`. By `Dst: TransmuteFrom`, the set of // `DV`-valid referents of `dst` is a superset of the set of `SV`-valid // referents of `src`. -// - Reverse transmutation: Since the underlying cast is size-preserving, `dst` -// addresses the same referent as `src`. By `Src: TransmuteFrom`, -// the set of `DV`-valid referents of `src` is a subset of the set of -// `SV`-valid referents of `dst`. +// - Preserve source validity: Since the underlying cast is size-preserving, +// `dst` addresses the same referent as `src`. By +// `Src: TransmuteFrom`, the set of `DV`-valid referents of `dst` +// is a subset of the set of `SV`-valid referents of `src`. // - No safe code, given access to `src` and `dst`, can cause undefined // behavior: By `Dst: MutationCompatible`, at least one of // the following holds: // - `A` is `Exclusive` // - `Src: Immutable` and `Dst: Immutable` -// - `Dst: InvariantsEq`, which guarantees that `Src` and `Dst` have the -// same invariants, and permit interior mutation on the same byte ranges +// - `Dst: SharedCompatible`, which guarantees that it is sound for safe +// code to operate on `&Src` and `&Dst` pointing to the same byte range at +// the same time unsafe impl TryTransmuteFromPtr for Dst where @@ -162,9 +163,9 @@ where } // SAFETY: -// - Forwards transmutation: Since aliasing is `Shared` and `Src: Immutable`, +// - Preserve destination validity: Since aliasing is `Shared` and `Src: Immutable`, // `src` does not permit mutation of its referent. -// - Reverse transmutation: Since aliasing is `Shared` and `Dst: Immutable`, +// - Preserve source validity: Since aliasing is `Shared` and `Dst: Immutable`, // `dst` does not permit mutation of its referent. // - No safe code, given access to `src` and `dst`, can cause undefined // behavior: `Src: Immutable` and `Dst: Immutable` @@ -187,9 +188,9 @@ where /// /// At least one of the following must hold: /// - `Src: Read` and `Self: Read` -/// - `Self: InvariantsEq`, and, for some `V`: -/// - `Dst: TransmuteFrom` -/// - `Src: TransmuteFrom` +/// - `Self: SharedCompatible`, and, for some `V`: +/// - `Self: TransmuteFrom` +/// - `Src: TransmuteFrom` pub unsafe trait MutationCompatible {} #[allow(missing_copy_implementations, missing_debug_implementations)] @@ -204,38 +205,39 @@ where { } -/// Denotes that two types have the same invariants. +/// Denotes that shared references to two types may safely coexist on the same +/// referent. /// /// # Safety /// /// It is sound for safe code to operate on a `&T` and a `&Self` pointing to the /// same referent at the same time - no such safe code can cause undefined /// behavior. -pub unsafe trait InvariantsEq {} +pub unsafe trait SharedCompatible {} // SAFETY: Trivially sound to have multiple `&T` pointing to the same referent. -unsafe impl InvariantsEq for T {} +unsafe impl SharedCompatible for T {} -// SAFETY: `Dst: InvariantsEq + TransmuteFrom`, and `Src: +// SAFETY: `Dst: SharedCompatible + TransmuteFrom`, and `Src: // TransmuteFrom`. unsafe impl - MutationCompatible for Dst + MutationCompatible for Dst where Src: TransmuteFrom, - Dst: TransmuteFrom + InvariantsEq, + Dst: TransmuteFrom + SharedCompatible, { } #[allow(missing_debug_implementations, missing_copy_implementations)] -pub enum BecauseInvariantsEq {} +pub enum BecauseSharedCompatible {} -macro_rules! unsafe_impl_invariants_eq { +macro_rules! unsafe_impl_shared_compatible { ($tyvar:ident => $t:ty, $u:ty) => {{ crate::util::macros::__unsafe(); // SAFETY: The caller promises that this is sound. - unsafe impl<$tyvar> InvariantsEq<$t> for $u {} + unsafe impl<$tyvar> SharedCompatible<$t> for $u {} // SAFETY: The caller promises that this is sound. - unsafe impl<$tyvar> InvariantsEq<$u> for $t {} + unsafe impl<$tyvar> SharedCompatible<$u> for $t {} }}; } @@ -253,11 +255,11 @@ impl_transitive_transmute_from!(T => Wrapping => T => MaybeUninit); // validity as `T` // // [2] https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html#impl-Deref-for-ManuallyDrop%3CT%3E -unsafe impl InvariantsEq for ManuallyDrop {} +unsafe impl SharedCompatible for ManuallyDrop {} // SAFETY: See previous safety comment. -unsafe impl InvariantsEq> for T {} +unsafe impl SharedCompatible> for T {} -/// Transmutations which are always sound. +/// Exact pointer reinterpretations which are always sound. /// /// `TransmuteFromPtr` is a shorthand for [`TryTransmuteFromPtr`] and /// [`TransmuteFrom`]. @@ -293,8 +295,12 @@ where { } -/// Denotes that any `SV`-valid `Src` may soundly be transmuted into a -/// `DV`-valid `Self`. +/// Denotes that, for equally-sized referents, every bit pattern allowed for an +/// `SV`-valid `Src` is also allowed for a `DV`-valid `Self`. +/// +/// `TransmuteFrom` does not by itself authorize a pointer transmutation. It +/// provides a directional validity implication used by the pointer-transmutation +/// machinery. /// /// # Safety /// @@ -333,7 +339,7 @@ impl SizeEq for T { // SAFETY: Since `Src: IntoBytes`, the set of valid `Src`'s is the set of // initialized bit patterns, which is exactly the set allowed in the referent of // any `Initialized` `Ptr`. -unsafe impl TransmuteFrom for Dst +unsafe impl TransmuteFrom for Dst where Src: IntoBytes + ?Sized, Dst: ?Sized, @@ -341,18 +347,19 @@ where } // SAFETY: Since `Dst: FromBytes`, any initialized bit pattern may appear in the -// referent of a `Ptr`. This is exactly equal to the set of +// referent of a `Ptr`. This is exactly equal to the set of // bit patterns which may appear in the referent of any `Initialized` `Ptr`. -unsafe impl TransmuteFrom for Dst +unsafe impl TransmuteFrom for Dst where Src: ?Sized, Dst: FromBytes + ?Sized, { } -// FIXME(#2354): This seems like a smell - the soundness of this bound has -// nothing to do with `Src` or `Dst` - we're basically just saying `[u8; N]` is -// transmutable into `[u8; N]`. +// FIXME(#2354): This relation does not depend on `Src` or `Dst`. That is +// expected semantically: `Initialized` permits the same states regardless of +// referent type. The awkward part is encoding that validity-state relation as a +// trait relation between referent types. // SAFETY: The set of allowed bit patterns in the referent of any `Initialized` // `Ptr` is the same regardless of referent type. @@ -363,9 +370,10 @@ where { } -// FIXME(#2354): This seems like a smell - the soundness of this bound has -// nothing to do with `Dst` - we're basically just saying that any type is -// transmutable into `MaybeUninit<[u8; N]>`. +// FIXME(#2354): This relation does not depend on `Src` or `Dst`. That is +// expected semantically: every validity state implies `Uninit`, which permits +// any byte sequence. The awkward part is encoding that validity-state relation +// as a trait relation between referent types. // SAFETY: A `Dst` with validity `Uninit` permits any byte sequence, and // therefore can be transmuted from any value. @@ -400,7 +408,7 @@ const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T => Unalign) // sound for these two references to exist at the same time since it's already // possible for safe code to get into this state. #[allow(clippy::multiple_unsafe_ops_per_block)] -const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Unalign) }; +const _: () = unsafe { unsafe_impl_shared_compatible!(T => T, Unalign) }; // SAFETY: // - `Wrapping` has the same size as `T` [1]. @@ -433,7 +441,7 @@ const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T => Wrapping // already possible for safe code to obtain a `&Wrapping` and a `&T` pointing // to the same referent at the same time. Thus, this must be sound. #[allow(clippy::multiple_unsafe_ops_per_block)] -const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Wrapping) }; +const _: () = unsafe { unsafe_impl_shared_compatible!(T => T, Wrapping) }; // SAFETY: // - `UnsafeCell` has the same size as `T` [1]. @@ -478,7 +486,7 @@ impl_transitive_transmute_from!(T: ?Sized => UnsafeCell => T => Cell); // explicitly guaranteed, but it's obvious from `MaybeUninit`'s documentation // that this is the intention: // https://doc.rust-lang.org/1.85.0/core/mem/union.MaybeUninit.html -unsafe impl TransmuteFrom for MaybeUninit {} +unsafe impl TransmuteFrom for MaybeUninit {} impl SizeEq for MaybeUninit { type CastFrom = CastSizedExact; diff --git a/rust/zerocopy/src/ref.rs b/rust/zerocopy/src/ref.rs index e49f2a887ffa74..7a8876a0587bbf 100644 --- a/rust/zerocopy/src/ref.rs +++ b/rust/zerocopy/src/ref.rs @@ -10,7 +10,7 @@ // those terms. use super::*; use crate::pointer::{ - BecauseInvariantsEq, BecauseMutationCompatible, MutationCompatible, TransmuteFromPtr, + BecauseMutationCompatible, BecauseSharedCompatible, MutationCompatible, TransmuteFromPtr, }; mod def { @@ -18,7 +18,7 @@ mod def { use crate::{ ByteSlice, ByteSliceMut, CloneableByteSlice, CopyableByteSlice, IntoByteSlice, - IntoByteSliceMut, + IntoByteSliceMut, KnownLayout, PointerMetadata, }; /// A typed reference derived from a byte slice. @@ -59,30 +59,83 @@ mod def { /// } /// } /// ``` - pub struct Ref( - // INVARIANTS: The referent (via `.deref`, `.deref_mut`, `.into`) byte - // slice is aligned to `T`'s alignment and its size corresponds to a - // valid size for `T`. - B, - PhantomData, - ); - - impl Ref { + pub struct Ref { + // INVARIANTS: + // - The referent (via `.deref`, `.deref_mut`, `.into`) byte slice is + // aligned to `T`'s alignment, and its size is valid for `T`. + // - If `T: KnownLayout`, let `metadata` be + // `T::PointerMetadata::from_elem_count` applied to `elem_count`. + // Then `T::size_for_metadata(metadata)` is `Some(referent.len())`. + bytes: B, + // Ignored when `T` does not implement `KnownLayout`. + elem_count: usize, + _marker: PhantomData, + } + + impl Ref { + /// Constructs a new `Ref` to a sized `T`. + /// + /// # Safety + /// + /// `bytes` dereferences (via [`deref`], [`deref_mut`], and [`into`]) to + /// a byte slice which is aligned to `T`'s alignment and whose size is + /// `size_of::()`. + /// + /// [`deref`]: core::ops::Deref::deref + /// [`deref_mut`]: core::ops::DerefMut::deref_mut + /// [`into`]: core::convert::Into::into + pub(super) unsafe fn new_sized_unchecked(bytes: B) -> Ref { + // INVARIANTS: The caller has promised that `bytes`'s referent is + // validly aligned and has size `size_of::()`. If `T: + // KnownLayout`, then, since `T` is sized, `T::PointerMetadata` is + // `()`. `from_elem_count` returns `()` for any element count, and + // `T::size_for_metadata(()) == Some(size_of::())`. `size_of` + // reports this size in bytes [1]. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + Ref { bytes, elem_count: 0, _marker: PhantomData } + } + } + + impl Ref { /// Constructs a new `Ref`. /// /// # Safety /// /// `bytes` dereferences (via [`deref`], [`deref_mut`], and [`into`]) to - /// a byte slice which is aligned to `T`'s alignment and whose size is a - /// valid size for `T`. + /// a byte slice which is aligned to `T`'s alignment and satisfies + /// `T::size_for_metadata(metadata) == Some(referent.len())`. /// /// [`deref`]: core::ops::Deref::deref /// [`deref_mut`]: core::ops::DerefMut::deref_mut /// [`into`]: core::convert::Into::into - pub(crate) unsafe fn new_unchecked(bytes: B) -> Ref { + pub(super) unsafe fn new_unchecked(bytes: B, metadata: T::PointerMetadata) -> Ref { // INVARIANTS: The caller has promised that `bytes`'s referent is - // validly-aligned and has a valid size. - Ref(bytes, PhantomData) + // validly aligned and has exactly the size described by `metadata`. + // `KnownLayout::PointerMetadata` is `()` for sized types and + // `usize` for slice DSTs. For both permitted metadata types, + // `from_elem_count(metadata.to_elem_count()) == metadata`: `()` + // round-trips through zero, while `usize` round-trips through + // itself. Thus, storing the element count preserves the metadata + // whose size the caller validated. This agrees with Rust's + // language-level definition of slice pointer metadata [1]. + // + // [1] Per https://doc.rust-lang.org/1.56.0/reference/dynamically-sized-types.html: + // + // Pointers to slices also store the number of elements of the + // slice. + Ref { bytes, elem_count: metadata.to_elem_count(), _marker: PhantomData } + } + + /// Gets the pointer metadata represented by this `Ref`. + #[inline(always)] + pub(super) fn pointer_metadata(&self) -> T::PointerMetadata { + // INVARIANTS: `KnownLayout::PointerMetadata` is either `()` or + // `usize`. For `()`, `from_elem_count` ignores the element count; + // for `usize`, it returns the element count unchanged. + T::PointerMetadata::from_elem_count(self.elem_count) } } @@ -96,16 +149,18 @@ mod def { /// `Any::downcast_ref`). /// /// `as_byte_slice` promises to return a `ByteSlice` whose referent is - /// validly-aligned for `T` and has a valid size for `T`. + /// validly aligned for `T` and has a valid size for `T`. If `T: + /// KnownLayout`, then `T::size_for_metadata(self.pointer_metadata())` + /// returns `Some(referent.len())`. pub(crate) unsafe fn as_byte_slice(&self) -> &impl ByteSlice { // INVARIANTS: The caller promises not to call methods other than // those on `ByteSlice`. Since `B: ByteSlice`, dereference stability // guarantees that calling `ByteSlice` methods will not change the - // address or length of `self.0`'s referent. + // address or length of `self.bytes`'s referent. // - // SAFETY: By invariant on `self.0`, the alignment and size - // post-conditions are upheld. - &self.0 + // SAFETY: By invariant on `self`, the alignment, size, and metadata + // postconditions are upheld. + &self.bytes } } @@ -118,17 +173,20 @@ mod def { /// [`ByteSliceMut`] other than `ByteSliceMut` methods (for example, via /// `Any::downcast_mut`). /// - /// `as_byte_slice` promises to return a `ByteSlice` whose referent is - /// validly-aligned for `T` and has a valid size for `T`. + /// `as_byte_slice_mut` promises to return a `ByteSliceMut` whose + /// referent is validly aligned for `T` and has a valid size for `T`. If + /// `T: KnownLayout`, then + /// `T::size_for_metadata(self.pointer_metadata())` returns + /// `Some(referent.len())`. pub(crate) unsafe fn as_byte_slice_mut(&mut self) -> &mut impl ByteSliceMut { // INVARIANTS: The caller promises not to call methods other than // those on `ByteSliceMut`. Since `B: ByteSlice`, dereference // stability guarantees that calling `ByteSlice` methods will not - // change the address or length of `self.0`'s referent. + // change the address or length of `self.bytes`'s referent. // - // SAFETY: By invariant on `self.0`, the alignment and size - // post-conditions are upheld. - &mut self.0 + // SAFETY: By invariant on `self`, the alignment, size, and metadata + // postconditions are upheld. + &mut self.bytes } } @@ -141,17 +199,20 @@ mod def { /// [`IntoByteSlice`] other than `IntoByteSlice` methods (for example, /// via `Any::downcast_ref`). /// - /// `as_byte_slice` promises to return a `ByteSlice` whose referent is - /// validly-aligned for `T` and has a valid size for `T`. + /// `into_byte_slice` promises to return an `IntoByteSlice` whose + /// referent is validly aligned for `T` and has a valid size for `T`. If + /// `T: KnownLayout`, then + /// `T::size_for_metadata(self.pointer_metadata())` returns + /// `Some(referent.len())`. pub(crate) unsafe fn into_byte_slice(self) -> impl IntoByteSlice<'a> { // INVARIANTS: The caller promises not to call methods other than // those on `IntoByteSlice`. Since `B: ByteSlice`, dereference // stability guarantees that calling `ByteSlice` methods will not - // change the address or length of `self.0`'s referent. + // change the address or length of `self.bytes`'s referent. // - // SAFETY: By invariant on `self.0`, the alignment and size - // post-conditions are upheld. - self.0 + // SAFETY: By invariant on `self`, the alignment, size, and metadata + // postconditions are upheld. + self.bytes } } @@ -164,33 +225,38 @@ mod def { /// [`IntoByteSliceMut`] other than `IntoByteSliceMut` methods (for /// example, via `Any::downcast_mut`). /// - /// `as_byte_slice` promises to return a `ByteSlice` whose referent is - /// validly-aligned for `T` and has a valid size for `T`. + /// `into_byte_slice_mut` promises to return an `IntoByteSliceMut` whose + /// referent is validly aligned for `T` and has a valid size for `T`. If + /// `T: KnownLayout`, then + /// `T::size_for_metadata(self.pointer_metadata())` returns + /// `Some(referent.len())`. pub(crate) unsafe fn into_byte_slice_mut(self) -> impl IntoByteSliceMut<'a> { // INVARIANTS: The caller promises not to call methods other than // those on `IntoByteSliceMut`. Since `B: ByteSlice`, dereference // stability guarantees that calling `ByteSlice` methods will not - // change the address or length of `self.0`'s referent. + // change the address or length of `self.bytes`'s referent. // - // SAFETY: By invariant on `self.0`, the alignment and size - // post-conditions are upheld. - self.0 + // SAFETY: By invariant on `self`, the alignment, size, and metadata + // postconditions are upheld. + self.bytes } } impl Clone for Ref { #[inline] fn clone(&self) -> Ref { - // INVARIANTS: Since `B: CloneableByteSlice`, `self.0.clone()` has - // the same address and length as `self.0`. Since `self.0` upholds - // the field invariants, so does `self.0.clone()`. - Ref(self.0.clone(), PhantomData) + // INVARIANTS: Since `B: CloneableByteSlice`, `self.bytes.clone()` has + // the same address and length as `self.bytes`. We copy + // `self.elem_count`, so the clone also has the same metadata. Since + // `self` upholds the field invariants, so does the clone. + Ref { bytes: self.bytes.clone(), elem_count: self.elem_count, _marker: PhantomData } } } - // INVARIANTS: Since `B: CopyableByteSlice`, the copied `Ref`'s `.0` has the - // same address and length as the original `Ref`'s `.0`. Since the original - // upholds the field invariants, so does the copy. + // INVARIANTS: Since `B: CopyableByteSlice`, the copied `Ref`'s `bytes` has the + // same address and length as the original `Ref`'s `bytes`. The copied + // `elem_count` has the same metadata. Since the original upholds the field + // invariants, so does the copy. impl Copy for Ref {} } @@ -198,7 +264,7 @@ mod def { pub use def::Ref; use crate::pointer::{ - invariant::{Aligned, BecauseExclusive, Initialized, Unaligned, Valid}, + invariant::{Aligned, BecauseExclusive, Initialized, Safe, Unaligned}, BecauseRead, PtrInner, }; @@ -215,8 +281,14 @@ where return Err(err.with_src(bytes).into()); } - // SAFETY: We just validated size and alignment. - Ok(unsafe { Ref::new_unchecked(bytes) }) + // SAFETY: `size_of` returns `T`'s size in bytes [1]. The equality check + // above therefore establishes that `bytes` has exactly that byte + // length, and `validate_aligned_to` establishes alignment. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + Ok(unsafe { Ref::new_sized_unchecked(bytes) }) } } @@ -238,10 +310,15 @@ where )?; // SAFETY: We just validated alignment and that `bytes` is at least as // large as `T`. `bytes.split_at(mem::size_of::())?` ensures that the - // new `bytes` is exactly the size of `T`. By safety postcondition on + // new `bytes` is exactly the size of `T`, measured in bytes as specified + // for `size_of` [1]. By safety postcondition on // `SplitByteSlice::split_at` we can rely on `split_at` to produce the // correct `bytes` and `suffix`. - let r = unsafe { Ref::new_unchecked(bytes) }; + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + let r = unsafe { Ref::new_sized_unchecked(bytes) }; Ok((r, suffix)) } @@ -259,11 +336,16 @@ where } // SAFETY: Since `split_at` is defined as `bytes_len - size_of::()`, // the `bytes` which results from `let (prefix, bytes) = - // bytes.split_at(split_at)?` has length `size_of::()`. After - // constructing `bytes`, we validate that it has the proper alignment. - // By safety postcondition on `SplitByteSlice::split_at` we can rely on - // `split_at` to produce the correct `prefix` and `bytes`. - let r = unsafe { Ref::new_unchecked(bytes) }; + // bytes.split_at(split_at)?` has length `size_of::()`, measured in + // bytes as specified for `size_of` [1]. After constructing `bytes`, we + // validate that it has the proper alignment. By safety postcondition on + // `SplitByteSlice::split_at` we can rely on `split_at` to produce the + // correct `prefix` and `bytes`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + let r = unsafe { Ref::new_sized_unchecked(bytes) }; Ok((prefix, r)) } } @@ -306,17 +388,64 @@ where /// /// let _ = Ref::<_, ZSTy>::from_bytes(&b"UU"[..]); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_bytes", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[must_use = "has no side effects"] #[inline] pub fn from_bytes(source: B) -> Result, CastError> { static_assert_dst_is_not_zst!(T); - if let Err(e) = - Ptr::from_ref(source.deref()).try_cast_into_no_leftover::(None) + let metadata = match Ptr::from_ref(source.deref()) + .try_cast_into_no_leftover::(None) { - return Err(e.with_src(()).with_src(source)); - } - // SAFETY: `try_cast_into_no_leftover` validates size and alignment. - Ok(unsafe { Ref::new_unchecked(source) }) + Ok(ptr) => T::pointer_to_metadata(ptr.as_inner().as_ptr()), + Err(e) => return Err(e.with_src(()).with_src(source)), + }; + // SAFETY: By its safety postcondition, `try_cast_into_no_leftover` + // returned a `Ptr` aligned as required for references [2], over the + // same byte range as `source.deref()`. `pointer_to_metadata` returns + // the metadata stored in that pointer, and `size_for_metadata` computes + // the corresponding dynamically-known byte size [1]. Thus, + // `T::size_for_metadata(metadata)` is `Some(source.len())`. + // `ByteSlice` and its applicable subtraits guarantee that the stored + // `source` continues to dereference or convert to that address and + // length. These facts establish both preconditions of `new_unchecked`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html: + // + // Returns the size of the pointed-to value in bytes. + // + // ... when `T` has no statically-known size ... `size_of_val` can + // be used to get the dynamically-known size. + // + // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html: + // + // Returns the ABI-required minimum alignment of the type of the + // value that `val` points to. + // + // Every reference to a value of the type `T` must be a multiple of + // this number. + Ok(unsafe { Ref::new_unchecked(source, metadata) }) } } @@ -360,14 +489,37 @@ where /// /// let _ = Ref::<_, ZSTy>::from_prefix(&b"UU"[..]); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_prefix", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[must_use = "has no side effects"] #[inline] pub fn from_prefix(source: B) -> Result<(Ref, B), CastError> { static_assert_dst_is_not_zst!(T); - let remainder = match Ptr::from_ref(source.deref()) + let (metadata, remainder) = match Ptr::from_ref(source.deref()) .try_cast_into::(CastType::Prefix, None) { - Ok((_, remainder)) => remainder, + Ok((ptr, remainder)) => (T::pointer_to_metadata(ptr.as_inner().as_ptr()), remainder), Err(e) => { return Err(e.with_src(()).with_src(source)); } @@ -381,11 +533,35 @@ where #[allow(unstable_name_collisions)] let split_at = unsafe { source.len().unchecked_sub(remainder.len()) }; let (bytes, suffix) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?; - // SAFETY: `try_cast_into` validates size and alignment, and returns a - // `split_at` that indicates how many bytes of `source` correspond to a - // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we - // can rely on `split_at` to produce the correct `source` and `suffix`. - let r = unsafe { Ref::new_unchecked(bytes) }; + // SAFETY: By its safety postcondition, `try_cast_into` returned an + // `Ptr` aligned as required for references [2], and a remainder + // which are non-overlapping, cover `source.deref()`, and place the + // `Ptr` at `source.deref()`'s address. + // Therefore, `split_at` is the typed pointer's byte length. By the + // safety postcondition of `SplitByteSlice::split_at`, `bytes` has that + // length and address. `pointer_to_metadata` returns the metadata stored + // in the typed pointer, and `size_for_metadata` computes the + // corresponding dynamically-known byte size [1]. Thus, + // `T::size_for_metadata(metadata)` is `Some(bytes.len())`. The + // `SplitByteSlice` and `ByteSlice` contracts preserve the established + // address and length, satisfying both preconditions of + // `new_unchecked`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html: + // + // Returns the size of the pointed-to value in bytes. + // + // ... when `T` has no statically-known size ... `size_of_val` can + // be used to get the dynamically-known size. + // + // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html: + // + // Returns the ABI-required minimum alignment of the type of the + // value that `val` points to. + // + // Every reference to a value of the type `T` must be a multiple of + // this number. + let r = unsafe { Ref::new_unchecked(bytes, metadata) }; Ok((r, suffix)) } @@ -425,14 +601,37 @@ where /// /// let _ = Ref::<_, ZSTy>::from_suffix(&b"UU"[..]); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_suffix", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[must_use = "has no side effects"] #[inline] pub fn from_suffix(source: B) -> Result<(B, Ref), CastError> { static_assert_dst_is_not_zst!(T); - let remainder = match Ptr::from_ref(source.deref()) + let (metadata, remainder) = match Ptr::from_ref(source.deref()) .try_cast_into::(CastType::Suffix, None) { - Ok((_, remainder)) => remainder, + Ok((ptr, remainder)) => (T::pointer_to_metadata(ptr.as_inner().as_ptr()), remainder), Err(e) => { let e = e.with_src(()); return Err(e.with_src(source)); @@ -441,11 +640,39 @@ where let split_at = remainder.len(); let (prefix, bytes) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?; - // SAFETY: `try_cast_into` validates size and alignment, and returns a - // `split_at` that indicates how many bytes of `source` correspond to a - // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we - // can rely on `split_at` to produce the correct `prefix` and `bytes`. - let r = unsafe { Ref::new_unchecked(bytes) }; + // SAFETY: By its safety postcondition, `try_cast_into` returned an + // `Ptr` aligned as required for references [2], and a remainder + // which are non-overlapping, cover `source.deref()`, and place the + // remainder at `source.deref()`'s address. More precisely, + // `PtrInner::try_cast_into` constructs the typed suffix from the + // `r_slice` produced by `split_at_unchecked(split_at)` and returns its + // `l_slice` as `remainder`. Thus, even when `T` is zero-sized, the typed + // pointer and the `bytes` produced by splitting `source` at + // `remainder.len()` have the same address and length. By the safety + // postcondition of `SplitByteSlice::split_at`, `bytes` retains that + // address and length. `pointer_to_metadata` returns the metadata stored + // in the typed pointer, and `size_for_metadata` computes the + // corresponding dynamically-known byte size [1]. Thus, + // `T::size_for_metadata(metadata)` is `Some(bytes.len())`. The + // `SplitByteSlice` and `ByteSlice` contracts preserve the established + // address and length, satisfying both preconditions of + // `new_unchecked`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html: + // + // Returns the size of the pointed-to value in bytes. + // + // ... when `T` has no statically-known size ... `size_of_val` can + // be used to get the dynamically-known size. + // + // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html: + // + // Returns the ABI-required minimum alignment of the type of the + // value that `val` points to. + // + // Every reference to a value of the type `T` must be a multiple of + // this number. + let r = unsafe { Ref::new_unchecked(bytes, metadata) }; Ok((prefix, r)) } } @@ -487,6 +714,24 @@ where /// /// let _ = Ref::<_, ZSTy>::from_bytes_with_elems(&b"UU"[..], 42); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_bytes_with_elems", + format = "coco", + arity = 2, + [ + open + @index 1 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 2 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] pub fn from_bytes_with_elems(source: B, count: usize) -> Result, CastError> { static_assert_dst_is_not_zst!(T); @@ -497,7 +742,37 @@ where if source.len() != expected_len { return Err(SizeError::new(source).into()); } - Self::from_bytes(source) + if let Err(e) = Ptr::from_ref(source.deref()) + .try_cast_into_no_leftover::(Some(count)) + { + return Err(e.with_src(()).with_src(source)); + } + // SAFETY: `size_for_metadata(count)` returned `expected_len`, and the + // equality check established that this is `source.len()`. This is the + // dynamically-known byte size described by `size_of_val` [1]. By its + // safety postcondition, `try_cast_into_no_leftover(Some(count))` only + // succeeded after producing a `Ptr` aligned as required for + // references [2], with that metadata over the same byte range as + // `source.deref()`. `ByteSlice` and its applicable subtraits guarantee + // that the stored `source` continues to dereference or convert to that + // address and length. These facts establish both preconditions of + // `new_unchecked`. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html: + // + // Returns the size of the pointed-to value in bytes. + // + // ... when `T` has no statically-known size ... `size_of_val` can + // be used to get the dynamically-known size. + // + // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html: + // + // Returns the ABI-required minimum alignment of the type of the + // value that `val` points to. + // + // Every reference to a value of the type `T` must be a multiple of + // this number. + Ok(unsafe { Ref::new_unchecked(source, count) }) } } @@ -538,6 +813,24 @@ where /// /// let _ = Ref::<_, ZSTy>::from_prefix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_prefix_with_elems", + format = "coco", + arity = 2, + [ + open + @index 1 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 2 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] pub fn from_prefix_with_elems( source: B, @@ -549,7 +842,7 @@ where None => return Err(SizeError::new(source).into()), }; let (prefix, bytes) = source.split_at(expected_len).map_err(SizeError::new)?; - Self::from_bytes(prefix).map(move |l| (l, bytes)) + Self::from_bytes_with_elems(prefix, count).map(move |l| (l, bytes)) } /// Constructs a `Ref` from the suffix of the given bytes with DST length @@ -584,6 +877,24 @@ where /// /// let _ = Ref::<_, ZSTy>::from_suffix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error! /// ``` + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_from_suffix_with_elems", + format = "coco", + arity = 2, + [ + open + @index 1 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 2 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] pub fn from_suffix_with_elems( source: B, @@ -602,7 +913,7 @@ where // SAFETY: The preceding `source.len().checked_sub(expected_len)` // guarantees that `split_at` is in-bounds. let (bytes, suffix) = unsafe { source.split_at_unchecked(split_at) }; - Self::from_bytes(suffix).map(move |l| (bytes, l)) + Self::from_bytes_with_elems(suffix, count).map(move |l| (bytes, l)) } } @@ -618,12 +929,37 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::into_ref(r)` instead of `r.into_ref()`. This is so that /// there is no conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_into_ref", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[must_use = "has no side effects"] #[inline(always)] pub fn into_ref(r: Self) -> &'a T { // Presumably unreachable, since we've guarded each constructor of `Ref`. static_assert_dst_is_not_zst!(T); + let metadata = r.pointer_metadata(); + // SAFETY: We don't call any methods on `b` other than those provided by // `IntoByteSlice`. let b = unsafe { r.into_byte_slice() }; @@ -642,12 +978,12 @@ where return ptr.as_ref(); } - // PANICS: By post-condition on `into_byte_slice`, `b`'s size and - // alignment are valid for `T`. By post-condition, `b.into_byte_slice()` - // produces a byte slice with identical address and length to that - // produced by `b.deref()`. + // PANICS: By post-condition on `into_byte_slice`, `b` is aligned for + // `T`, and its size is exactly the size encoded by `metadata`. By + // post-condition, `b.into_byte_slice()` produces a byte slice with + // identical address and length to that produced by `b.deref()`. let ptr = Ptr::from_ref(b.into_byte_slice()) - .try_cast_into_no_leftover::(None) + .try_cast_into_no_leftover::(Some(metadata)) .expect("zerocopy internal error: into_ref should be infallible"); let ptr = ptr.recall_validity(); ptr.as_ref() @@ -666,12 +1002,32 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::into_mut(r)` instead of `r.into_mut()`. This is so that /// there is no conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_into_mut", + format = "coco", + arity = 2, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ] + )] #[must_use = "has no side effects"] #[inline(always)] pub fn into_mut(r: Self) -> &'a mut T { // Presumably unreachable, since we've guarded each constructor of `Ref`. static_assert_dst_is_not_zst!(T); + let metadata = r.pointer_metadata(); + // SAFETY: We don't call any methods on `b` other than those provided by // `IntoByteSliceMut`. let b = unsafe { r.into_byte_slice_mut() }; @@ -686,7 +1042,7 @@ where T, _, (BecauseRead, BecauseExclusive), - (BecauseMutationCompatible, BecauseInvariantsEq), + (BecauseMutationCompatible, BecauseSharedCompatible), >(ptr) }; @@ -697,13 +1053,13 @@ where return ptr.as_mut(); } - // PANICS: By post-condition on `into_byte_slice_mut`, `b`'s size and - // alignment are valid for `T`. By post-condition, - // `b.into_byte_slice_mut()` produces a byte slice with identical - // address and length to that produced by `b.deref_mut()`. + // PANICS: By post-condition on `into_byte_slice_mut`, `b` is aligned + // for `T`, and its size is exactly the size encoded by `metadata`. By + // post-condition, `b.into_byte_slice_mut()` produces a byte slice with + // identical address and length to that produced by `b.deref_mut()`. let ptr = Ptr::from_mut(b.into_byte_slice_mut()) - .try_cast_into_no_leftover::(None) - .expect("zerocopy internal error: into_ref should be infallible"); + .try_cast_into_no_leftover::(Some(metadata)) + .expect("zerocopy internal error: into_mut should be infallible"); let ptr = ptr.recall_validity::<_, (_, (_, _))>(); ptr.as_mut() } @@ -719,6 +1075,29 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::bytes(r)` instead of `r.bytes()`. This is so that there is /// no conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_bytes", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] pub fn bytes(r: &Self) -> &[u8] { // SAFETY: We don't call any methods on `b` other than those provided by @@ -737,6 +1116,29 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::bytes_mut(r)` instead of `r.bytes_mut()`. This is so that /// there is no conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_bytes_mut", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] pub fn bytes_mut(r: &mut Self) -> &mut [u8] { // SAFETY: We don't call any methods on `b` other than those provided by @@ -755,6 +1157,12 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::read(r)` instead of `r.read()`. This is so that there is no /// conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_read_static_size", + format = "coco_static_size", + )] #[must_use = "has no side effects"] #[inline] pub fn read(r: &Self) -> T { @@ -780,14 +1188,20 @@ where /// Note: this is an associated function, which means that you have to call /// it as `Ref::write(r, t)` instead of `r.write(t)`. This is so that there /// is no conflict with a method on the inner type. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_write_static_size", + format = "coco_static_size", + )] #[inline] pub fn write(r: &mut Self, t: T) { // SAFETY: We don't call any methods on `b` other than those provided by // `ByteSliceMut`. let b = unsafe { r.as_byte_slice_mut() }; - // SAFETY: By postcondition on `as_byte_slice_mut`, we know that `b` is - // a valid size and alignment for `T`. By safety invariant on + // SAFETY: By postcondition on `as_byte_slice_mut`, we know that `b` is a + // valid size and alignment for `T`. By safety invariant on // `ByteSlice`, we know that this is preserved via `.deref()`. Writing // `t` to the buffer will allow all of the bytes of `t` to be accessed // as a `[u8]`, but because `T: IntoBytes`, we know that this is sound. @@ -801,11 +1215,38 @@ where T: FromBytes + KnownLayout + Immutable + ?Sized, { type Target = T; + + /// Dereferences the value. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_deref", + format = "coco", + arity = 3, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ], + [ + @index 3 + @title "Dynamically Padded" + @variant "dynamic_padding" + ] + )] #[inline] fn deref(&self) -> &T { // Presumably unreachable, since we've guarded each constructor of `Ref`. static_assert_dst_is_not_zst!(T); + let metadata = self.pointer_metadata(); + // SAFETY: We don't call any methods on `b` other than those provided by // `ByteSlice`. let b = unsafe { self.as_byte_slice() }; @@ -824,11 +1265,12 @@ where return ptr.as_ref(); } - // PANICS: By postcondition on `as_byte_slice`, `b`'s size and alignment - // are valid for `T`, and by invariant on `ByteSlice`, these are - // preserved through `.deref()`, so this `unwrap` will not panic. + // PANICS: By postcondition on `as_byte_slice`, `b` is aligned for `T`, + // and its size is exactly the size encoded by `metadata`. By invariant + // on `ByteSlice`, these are preserved through `.deref()`, so this + // `unwrap` will not panic. let ptr = Ptr::from_ref(b) - .try_cast_into_no_leftover::(None) + .try_cast_into_no_leftover::(Some(metadata)) .expect("zerocopy internal error: Deref::deref should be infallible"); let ptr = ptr.recall_validity(); ptr.as_ref() @@ -843,11 +1285,32 @@ where // add a separate inherent method for this? T: FromBytes + IntoBytes + KnownLayout + Immutable + ?Sized, { + /// Mutably dereferences the value. + /// + #[doc = codegen_section!( + header = "h5", + bench = "ref_type_deref_mut", + format = "coco", + arity = 2, + [ + open + @index 1 + @title "Sized" + @variant "static_size" + ], + [ + @index 2 + @title "Unsized" + @variant "dynamic_size" + ] + )] #[inline] fn deref_mut(&mut self) -> &mut T { // Presumably unreachable, since we've guarded each constructor of `Ref`. static_assert_dst_is_not_zst!(T); + let metadata = self.pointer_metadata(); + // SAFETY: We don't call any methods on `b` other than those provided by // `ByteSliceMut`. let b = unsafe { self.as_byte_slice_mut() }; @@ -862,7 +1325,7 @@ where T, _, (BecauseRead, BecauseExclusive), - (BecauseMutationCompatible, BecauseInvariantsEq), + (BecauseMutationCompatible, BecauseSharedCompatible), >(ptr) }; @@ -873,12 +1336,12 @@ where return ptr.as_mut(); } - // PANICS: By postcondition on `as_byte_slice_mut`, `b`'s size and - // alignment are valid for `T`, and by invariant on `ByteSlice`, these - // are preserved through `.deref_mut()`, so this `unwrap` will not - // panic. + // PANICS: By postcondition on `as_byte_slice_mut`, `b` is aligned for + // `T`, and its size is exactly the size encoded by `metadata`. By + // invariant on `ByteSlice`, these are preserved through `.deref_mut()`, + // so this `unwrap` will not panic. let ptr = Ptr::from_mut(b) - .try_cast_into_no_leftover::(None) + .try_cast_into_no_leftover::(Some(metadata)) .expect("zerocopy internal error: DerefMut::deref_mut should be infallible"); let ptr = ptr.recall_validity::<_, (_, (_, BecauseExclusive))>(); ptr.as_mut() @@ -958,13 +1421,13 @@ where /// `T: Sized` and `ptr`'s referent must have size `size_of::()`. #[inline(always)] unsafe fn cast_for_sized<'a, T, A, R, S>( - ptr: Ptr<'a, [u8], (A, Aligned, Valid)>, -) -> Ptr<'a, T, (A, Unaligned, Valid)> + ptr: Ptr<'a, [u8], (A, Aligned, Safe)>, +) -> Ptr<'a, T, (A, Unaligned, Safe)> where T: FromBytes + KnownLayout + ?Sized, A: crate::invariant::Aliasing, [u8]: MutationCompatible, - T: TransmuteFromPtr, + T: TransmuteFromPtr, { use crate::pointer::cast::{Cast, Project}; @@ -990,7 +1453,7 @@ where ptr.recall_validity::() .cast::<_, CastForSized, _>() - .recall_validity::() + .recall_validity::() } #[cfg(test)] @@ -1028,6 +1491,66 @@ mod tests { assert_eq!(buf_ptr, deref_ptr); } + #[test] + #[allow(clippy::clone_on_copy)] // Exercise `Ref`'s manual `Clone` impl. + fn test_with_elems_preserves_ambiguous_metadata() { + #[derive(FromBytes, Immutable, KnownLayout)] + #[repr(C, align(8))] + struct Frame { + header: [u8; 8], + body: [u8], + } + + #[derive(FromBytes, Immutable, KnownLayout)] + #[repr(C, align(8))] + struct OddHeader { + header: [u8; 7], + body: [u8], + } + + // Dynamic trailing padding makes these pairs of distinct element + // counts produce the same total byte size. Thus, the byte length alone + // cannot recover the count supplied to a `*_with_elems` constructor. + assert_eq!(Frame::size_for_metadata(1), Some(16)); + assert_eq!(Frame::size_for_metadata(8), Some(16)); + assert_eq!(OddHeader::size_for_metadata(0), Some(8)); + assert_eq!(OddHeader::size_for_metadata(1), Some(8)); + + let bytes = Align::<[u8; 32], AU64>::default(); + + let whole = Ref::<_, Frame>::from_bytes_with_elems(&bytes.t[..16], 1).unwrap(); + assert_eq!(whole.body.len(), 1); + assert_eq!(Ref::bytes(&whole).len(), 16); + assert_eq!(whole.clone().body.len(), 1); + let copied = whole; + assert_eq!(copied.body.len(), 1); + assert_eq!(Ref::into_ref(whole).body.len(), 1); + + let (prefix, rest) = Ref::<_, Frame>::from_prefix_with_elems(&bytes.t[..], 1).unwrap(); + assert_eq!(prefix.body.len(), 1); + assert_eq!(rest.len(), 16); + + let (before, suffix) = Ref::<_, Frame>::from_suffix_with_elems(&bytes.t[..24], 1).unwrap(); + assert_eq!(before.len(), 8); + assert_eq!(suffix.body.len(), 1); + + // Zero is real metadata, not a sentinel for "infer from the bytes". + let empty = Ref::<_, OddHeader>::from_bytes_with_elems(&bytes.t[..8], 0).unwrap(); + assert!(empty.body.is_empty()); + assert!(Ref::into_ref(empty).body.is_empty()); + + // The implicit constructors still choose the largest fitting count, + // and retain the metadata selected during their validating cast. + let inferred = Ref::<_, Frame>::from_bytes(&bytes.t[..16]).unwrap(); + assert_eq!(inferred.body.len(), 8); + let (inferred, rest) = Ref::<_, Frame>::from_prefix(&bytes.t[..16]).unwrap(); + assert_eq!(inferred.body.len(), 8); + assert!(rest.is_empty()); + let (before, inferred) = Ref::<_, Frame>::from_suffix(&bytes.t[..16]).unwrap(); + assert!(before.is_empty()); + assert_eq!(inferred.body.len(), 8); + } + // Verify that values written to a `Ref` are properly shared between the // typed and untyped representations, that reads via `deref` and `read` // behave the same, and that writes via `deref_mut` and `write` behave the @@ -1251,6 +1774,13 @@ mod tests { *rf = u64::MAX; assert_eq!(buf.t, [0xFF; 8]); + + // Exercise the unsized `into_mut` route, which must use the metadata + // retained by the constructor rather than infer fresh metadata. + let mut buf = Align::<[u8; 8], u16>::default(); + let r = Ref::<_, [u16]>::from_bytes_with_elems(&mut buf.t[..], 4).unwrap(); + let rf = Ref::into_mut(r); + assert_eq!(rf.len(), 4); } #[test] @@ -1325,8 +1855,13 @@ mod benches { let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap(); b.iter(move || { // SAFETY: The preceding `from_bytes` succeeded, and so we know that - // `buf` is validly-aligned and has the correct length. - let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) }; + // `buf` is validly aligned and has the `size_of::()` byte + // length required by `new_sized_unchecked` [1]. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + let r = unsafe { Ref::<&mut [u8], AU64>::new_sized_unchecked(&mut *buf) }; test::black_box(Ref::into_mut(test::black_box(r))); }); } @@ -1349,8 +1884,13 @@ mod benches { let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap(); b.iter(|| { // SAFETY: The preceding `from_bytes` succeeded, and so we know that - // `buf` is validly-aligned and has the correct length. - let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) }; + // `buf` is validly aligned and has the `size_of::()` byte + // length required by `new_sized_unchecked` [1]. + // + // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html: + // + // Returns the size of a type in bytes. + let r = unsafe { Ref::<&mut [u8], AU64>::new_sized_unchecked(&mut *buf) }; let mut temp = test::black_box(r); test::black_box(temp.deref_mut()); }); diff --git a/rust/zerocopy/src/split_at.rs b/rust/zerocopy/src/split_at.rs index d7778425a31d3c..4aeb602f69e086 100644 --- a/rust/zerocopy/src/split_at.rs +++ b/rust/zerocopy/src/split_at.rs @@ -10,13 +10,19 @@ // those terms. use super::*; -use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Shared, Valid}; +use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Safe, Shared}; /// Types that can be split in two. /// /// This trait generalizes Rust's existing support for splitting slices to /// support slices and slice-based dynamically-sized types ("slice DSTs"). /// +/// For a `repr(C, packed)` or `repr(C, packed(N))` struct, the derive wraps +/// the trailing field's [`Elem`](Self::Elem) in [`Unalign`]. The returned +/// slice can therefore be accessed even when packing misaligns the original +/// elements. Each packed struct in the trailing-field chain adds one wrapper; +/// unpacked structs and [`ManuallyDrop`] forward their trailing type's `Elem`. +/// /// # Implementation /// /// **Do not implement this trait yourself!** Instead, use @@ -38,6 +44,28 @@ use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Shared, Valid}; /// This derive performs a sophisticated, compile-time safety analysis to /// determine whether a type is `SplitAt`. /// +/// A packed struct exposes its trailing elements through `Unalign`: +/// +/// ``` +/// use zerocopy::{FromBytes, SplitAt, Unalign}; +/// # use zerocopy_derive::*; +/// +/// #[derive(FromBytes, IntoBytes, KnownLayout, SplitAt)] +/// #[repr(C, packed)] +/// struct Packet { +/// tag: u8, +/// words: [u16], +/// } +/// +/// let mut bytes = [0u8; 5]; +/// let packet = Packet::mut_from_bytes(&mut bytes[..]).unwrap(); +/// let (left, right): (&mut Packet, &mut [Unalign]) = +/// packet.split_at_mut(1).unwrap().via_into_bytes(); +/// left.tag = 1; +/// right[0] = Unalign::new(0x0202); +/// assert_eq!(bytes, [1, 0, 0, 2, 2]); +/// ``` +/// /// # Safety /// /// This trait does not convey any safety guarantees to code outside this crate. @@ -58,10 +86,20 @@ use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Shared, Valid}; )] // # Safety // -// The trailing slice is well-aligned for its element type. `Self` is `[T]`, or -// a `repr(C)` or `repr(transparent)` slice DST. +// `Self` is a slice, a `repr(C)` (possibly packed) or `repr(transparent)` slice +// DST, or `ManuallyDrop` for some `T: SplitAt`. +// +// `Self::Elem` has the same size, bit validity, and `UnsafeCell` coverage as +// the actual trailing element type. Access through either representation +// preserves the other's validity, including writes through mutable references +// and interior mutation through shared references. For every aligned `Self`, +// its trailing slice's address is aligned for `Self::Elem`. pub unsafe trait SplitAt: KnownLayout { - /// The element type of the trailing slice. + /// The element type exposed by the split's returned slice. + /// + /// For `[T]`, this is `T`. For a struct, this is the trailing field's `Elem`, + /// wrapped in [`Unalign`] if the struct is packed. Nested packed structs + /// produce nested `Unalign` wrappers. type Elem; #[doc(hidden)] @@ -188,10 +226,7 @@ pub unsafe trait SplitAt: KnownLayout { /// Attempts to split `self` in two. /// /// Returns `None` if `l_len` is greater than the length of `self`'s - /// trailing slice, or if the given `l_len` would result in [the trailing - /// padding](KnownLayout#slice-dst-layout) of the left portion overlapping - /// the right portion. - /// + /// trailing slice. /// /// # Examples /// @@ -249,7 +284,8 @@ pub unsafe trait SplitAt: KnownLayout { } } -// SAFETY: `[T]`'s trailing slice is `[T]`, which is trivially aligned. +// SAFETY: `[T]` exposes its own element type, preserving size, validity, and +// `UnsafeCell` coverage. An aligned slice is aligned for its elements. unsafe impl SplitAt for [T] { type Elem = T; @@ -262,6 +298,29 @@ unsafe impl SplitAt for [T] { } } +// SAFETY: `ManuallyDrop` has the same layout and bit validity as `T` [1]. +// Its `KnownLayout` implementation preserves `T`'s metadata and trailing-slice +// layout. Forwarding `T::Elem` therefore preserves its size, validity, and +// alignment guarantees. Its `UnsafeCell` coverage also matches `T`'s, as +// established for `ManuallyDrop` in `impls.rs`; accesses through the forwarded +// element representation preserve validity in both directions. +// +// [1] Per https://doc.rust-lang.org/1.93.0/std/mem/struct.ManuallyDrop.html: +// +// `ManuallyDrop` is guaranteed to have the same layout and bit validity as +// `T` +unsafe impl SplitAt for ManuallyDrop { + type Elem = T::Elem; + + #[inline] + #[allow(dead_code)] + fn only_derive_is_allowed_to_implement_this_trait() + where + Self: Sized, + { + } +} + /// A `T` that has been split into two possibly-overlapping parts. /// /// For some dynamically sized types, the padding that appears after the @@ -270,10 +329,11 @@ unsafe impl SplitAt for [T] { /// requires trailing padding, the trailing padding of the left part of the /// split `T` will overlap the right part. If `T` is a mutable reference or /// permits interior mutation, you must ensure that the left and right parts do -/// not overlap. You can do this at zero-cost using using -/// [`Self::via_immutable`], [`Self::via_into_bytes`], or -/// [`Self::via_unaligned`], or with a dynamic check by using -/// [`Self::via_runtime_check`]. +/// not overlap. Use [`Self::via_into_bytes`] or +/// [`Self::via_no_dynamic_padding`] to establish this without a runtime check, +/// or [`Self::via_runtime_check`] to check a particular split. Shared references +/// to an [`Immutable`] type may overlap; use [`Self::via_immutable`] in that +/// case. #[derive(Debug)] pub struct Split { /// A pointer to the source slice DST. @@ -304,7 +364,7 @@ where T: ?Sized + SplitAt, { #[inline(always)] - fn into_ptr(self) -> Split> { + fn into_ptr(self) -> Split> { let source = Ptr::from_ref(self.source); // SAFETY: `Ptr::from_ref(self.source)` points to exactly `self.source` // and thus maintains the invariants of `self` with respect to `l_len`. @@ -423,59 +483,100 @@ where (l.as_ref(), r.as_ref()) } - /// Produces the split parts of `self`, using [`Unaligned`] to ensure that - /// it is sound to have concurrent references to both parts. + /// Produces the split parts of `self` using [`Self::via_no_dynamic_padding`]. + #[deprecated(note = "use `Split::via_no_dynamic_padding` instead")] + #[doc(hidden)] + #[must_use = "has no side effects"] + #[inline(always)] + pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem]) + where + T: Unaligned, + { + self.via_no_dynamic_padding() + } + + /// Produces the split parts of `self` using a compile-time assertion to + /// ensure that it is unconditionally sound to have concurrent references to + /// both parts. + /// + /// When possible, you should prefer using [`Self::via_into_bytes`], which + /// ensures this safety property through regular trait bounds. /// /// # Examples /// + /// In the below example, a single byte of padding exists (on most + /// platforms) between `Packet<[u16]>`'s `length` and `body` fields, + /// precluding the use of [`Self::via_into_bytes`]. + /// /// ``` - /// use zerocopy::{SplitAt, FromBytes}; + /// use zerocopy::SplitAt; /// # use zerocopy_derive::*; /// - /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Unaligned)] + /// #[derive(SplitAt, KnownLayout)] /// #[repr(C)] - /// struct Packet { + /// struct Packet { /// length: u8, - /// body: [u8], + /// body: B, /// } /// - /// // These bytes encode a `Packet`. - /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..]; - /// - /// let packet = Packet::ref_from_bytes(bytes).unwrap(); + /// let packet: &Packet<[u16]> = &Packet { length: 4, body: [1u16, 2, 3, 4, 5, 6, 7, 8, 9] }; /// /// assert_eq!(packet.length, 4); /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]); /// - /// // Attempt to split `packet` at `length`. - /// let split = packet.split_at(packet.length as usize).unwrap(); + /// if let Some(split) = packet.split_at(packet.length as usize) { + /// // Every body length requires no trailing padding. + /// let (packet, rest) = split.via_no_dynamic_padding(); + /// assert_eq!(packet.length, 4); + /// assert_eq!(packet.body, [1, 2, 3, 4]); + /// assert_eq!(rest, [5, 6, 7, 8, 9]); + /// } else { + /// unreachable!("The packet's length field is within its body"); + /// } + /// ``` /// - /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to - /// // return concurrent references to `packet` and `rest`. - /// let (packet, rest) = split.via_unaligned(); + /// # Compile-Time Assertions /// - /// assert_eq!(packet.length, 4); - /// assert_eq!(packet.body, [1, 2, 3, 4]); - /// assert_eq!(rest, [5, 6, 7, 8, 9]); + /// This method rejects at compile-time any type that admits dynamic + /// trailing padding. In the below example, some `split_at` indices would + /// produce left parts whose trailing padding overlaps with the right part: + /// + /// ```compile_fail,E0080 + /// use zerocopy::SplitAt; + /// # use zerocopy_derive::*; + /// + /// #[derive(SplitAt, KnownLayout)] + /// #[repr(C, align(2))] + /// struct Packet { + /// length: u16, + /// body: B, + /// } + /// + /// let packet: &Packet<[u8]> = &Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] }; + /// + /// if let Some(split) = packet.split_at(packet.length as usize) { + /// let _ = split.via_no_dynamic_padding(); // ⚠ Compile Error! + /// } else { + /// unreachable!("The packet's length field is within its body"); + /// } /// ``` /// - #[doc = codegen_header!("h5", "split_via_unaligned")] + /// If you need to split such types, use [`Self::via_runtime_check`]. + /// + #[doc = codegen_header!("h5", "split_via_no_dynamic_padding")] /// /// See [`Split::via_immutable`](#method.split_via_immutable.codegen). #[must_use = "has no side effects"] #[inline(always)] - pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem]) - where - T: Unaligned, - { - let (l, r) = self.into_ptr().via_unaligned(); + pub fn via_no_dynamic_padding(self) -> (&'a T, &'a [T::Elem]) { + let (l, r) = self.into_ptr().via_no_dynamic_padding(); (l.as_ref(), r.as_ref()) } /// Produces the split parts of `self`, using a dynamic check to ensure that /// it is sound to have concurrent references to both parts. You should /// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or - /// [`Self::via_unaligned`], which have no runtime cost. + /// [`Self::via_no_dynamic_padding`], which perform no runtime check. /// /// Note that this check is overly conservative if `T` is [`Immutable`]; for /// some types, this check will reject some splits which @@ -484,52 +585,40 @@ where /// # Examples /// /// ``` - /// use zerocopy::{SplitAt, FromBytes, IntoBytes, network_endian::U16}; + /// use zerocopy::SplitAt; /// # use zerocopy_derive::*; /// - /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Debug)] + /// #[derive(SplitAt, KnownLayout)] /// #[repr(C, align(2))] - /// struct Packet { - /// length: U16, - /// body: [u8], + /// struct Packet { + /// length: u16, + /// body: B, /// } /// - /// // These bytes encode a `Packet`. - /// let bytes = [ - /// 4u16.to_be(), - /// 1u16.to_be(), - /// 2u16.to_be(), - /// 3u16.to_be(), - /// 4u16.to_be() - /// ]; - /// - /// let packet = Packet::ref_from_bytes(bytes.as_bytes()).unwrap(); + /// let packet: &Packet<[u8]> = &Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] }; /// /// assert_eq!(packet.length, 4); - /// assert_eq!(packet.body, [0, 1, 0, 2, 0, 3, 0, 4]); - /// - /// // Attempt to split `packet` at `length`. - /// let split = packet.split_at(packet.length.into()).unwrap(); + /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]); /// - /// // Use a dynamic check to prove that it's okay to return concurrent - /// // references to `packet` and `rest`. - /// let (packet, rest) = split.via_runtime_check().unwrap(); + /// // The two-byte prefix and four-byte body need no trailing padding. + /// if let Some(split) = packet.split_at(packet.length as usize) { + /// if let Ok((packet, rest)) = split.via_runtime_check() { + /// assert_eq!(packet.length, 4); + /// assert_eq!(packet.body, [1, 2, 3, 4]); + /// assert_eq!(rest, [5, 6, 7, 8, 9]); + /// } else { + /// unreachable!("A four-byte body requires no trailing padding"); + /// } + /// } else { + /// unreachable!("The packet's length field is within its body"); + /// } /// - /// assert_eq!(packet.length, 4); - /// assert_eq!(packet.body, [0, 1, 0, 2]); - /// assert_eq!(rest, [0, 3, 0, 4]); - /// - /// // Attempt to split `packet` at `length - 1`. - /// let idx = packet.length.get() - 1; - /// let split = packet.split_at(idx as usize).unwrap(); - /// - /// // Attempt (and fail) to use a dynamic check to prove that it's okay - /// // to return concurrent references to `packet` and `rest`. Note that - /// // this is a case of `via_runtime_check` being overly conservative. - /// // Although the left and right parts indeed overlap, the `Immutable` - /// // bound ensures that concurrently referencing these overlapping - /// // parts is sound. - /// assert!(split.via_runtime_check().is_err()); + /// // A three-byte body needs a padding byte that would overlap `rest`. + /// if let Some(split) = packet.split_at((packet.length - 1) as usize) { + /// assert!(split.via_runtime_check().is_err()); + /// } else { + /// unreachable!("One less than the packet's length is within its body"); + /// } /// ``` /// #[doc = codegen_section!( @@ -602,7 +691,7 @@ where T: ?Sized + SplitAt, { #[inline(always)] - fn into_ptr(self) -> Split> { + fn into_ptr(self) -> Split> { let source = Ptr::from_mut(self.source); // SAFETY: `Ptr::from_mut(self.source)` points to exactly `self.source`, // and thus maintains the invariants of `self` with respect to `l_len`. @@ -665,101 +754,135 @@ where (l.as_mut(), r.as_mut()) } - /// Produces the split parts of `self`, using [`Unaligned`] to ensure that - /// it is sound to have concurrent references to both parts. + /// Produces the split parts of `self` using [`Self::via_no_dynamic_padding`]. + #[deprecated(note = "use `Split::via_no_dynamic_padding` instead")] + #[doc(hidden)] + #[must_use = "has no side effects"] + #[inline(always)] + pub fn via_unaligned(self) -> (&'a mut T, &'a mut [T::Elem]) + where + T: Unaligned, + { + self.via_no_dynamic_padding() + } + + /// Produces the split parts of `self` using a compile-time assertion to + /// ensure that it is unconditionally sound to have concurrent references to + /// both parts. + /// + /// When possible, you should prefer using [`Self::via_into_bytes`], which + /// ensures this safety property through regular trait bounds. /// /// # Examples /// + /// In the below example, a single byte of padding exists (on most + /// platforms) between `Packet<[u16]>`'s `length` and `body` fields, + /// precluding the use of [`Self::via_into_bytes`]. + /// /// ``` - /// use zerocopy::{SplitAt, FromBytes}; + /// use zerocopy::SplitAt; /// # use zerocopy_derive::*; /// - /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Unaligned)] + /// #[derive(SplitAt, KnownLayout)] /// #[repr(C)] /// struct Packet { /// length: u8, /// body: B, /// } /// - /// // These bytes encode a `Packet`. - /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..]; - /// - /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap(); + /// let packet: &mut Packet<[u16]> = &mut Packet { length: 4, body: [1u16, 2, 3, 4, 5, 6, 7, 8, 9] }; /// /// assert_eq!(packet.length, 4); /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]); /// - /// { - /// // Attempt to split `packet` at `length`. - /// let split = packet.split_at_mut(packet.length as usize).unwrap(); - /// - /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to - /// // return concurrent references to `packet` and `rest`. - /// let (packet, rest) = split.via_unaligned(); - /// + /// // Every body length requires no trailing padding. + /// if let Some(split) = packet.split_at_mut(packet.length as usize) { + /// let (packet, rest) = split.via_no_dynamic_padding(); /// assert_eq!(packet.length, 4); /// assert_eq!(packet.body, [1, 2, 3, 4]); /// assert_eq!(rest, [5, 6, 7, 8, 9]); - /// /// rest.fill(0); + /// } else { + /// unreachable!("The packet's length field is within its body"); /// } /// /// assert_eq!(packet.length, 4); /// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]); /// ``` /// + /// # Compile-Time Assertions + /// + /// This method rejects at compile-time any type that admits dynamic + /// trailing padding. In the below example, some `split_at` indices would + /// produce left parts whose trailing padding overlaps with the right part: + /// + /// ```compile_fail,E0080 + /// use zerocopy::SplitAt; + /// # use zerocopy_derive::*; + /// + /// #[derive(SplitAt, KnownLayout)] + /// #[repr(C, align(2))] + /// struct Packet { + /// length: u16, + /// body: B, + /// } + /// + /// let packet: &mut Packet<[u8]> = &mut Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] }; + /// + /// if let Some(split) = packet.split_at_mut(packet.length as usize) { + /// let _ = split.via_no_dynamic_padding(); // ⚠ Compile Error! + /// } else { + /// unreachable!("The packet's length field is within its body"); + /// } + /// ``` + /// + /// If you need to split such types, use [`Self::via_runtime_check`]. + /// /// # Code Generation /// /// See [`Split::via_immutable`](#method.split_via_immutable.codegen). #[must_use = "has no side effects"] #[inline(always)] - pub fn via_unaligned(self) -> (&'a mut T, &'a mut [T::Elem]) - where - T: Unaligned, - { - let (l, r) = self.into_ptr().via_unaligned(); + pub fn via_no_dynamic_padding(self) -> (&'a mut T, &'a mut [T::Elem]) { + let (l, r) = self.into_ptr().via_no_dynamic_padding(); (l.as_mut(), r.as_mut()) } /// Produces the split parts of `self`, using a dynamic check to ensure that /// it is sound to have concurrent references to both parts. You should - /// prefer using [`Self::via_into_bytes`] or [`Self::via_unaligned`], which - /// have no runtime cost. + /// prefer using [`Self::via_into_bytes`] or + /// [`Self::via_no_dynamic_padding`], which perform no runtime check. /// /// # Examples /// /// ``` - /// use zerocopy::{SplitAt, FromBytes}; + /// use zerocopy::SplitAt; /// # use zerocopy_derive::*; /// - /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Debug)] - /// #[repr(C)] + /// #[derive(SplitAt, KnownLayout)] + /// #[repr(C, align(2))] /// struct Packet { - /// length: u8, + /// length: u16, /// body: B, /// } /// - /// // These bytes encode a `Packet`. - /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..]; - /// - /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap(); + /// let packet: &mut Packet<[u8]> = &mut Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] }; /// /// assert_eq!(packet.length, 4); /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]); /// - /// { - /// // Attempt to split `packet` at `length`. - /// let split = packet.split_at_mut(packet.length as usize).unwrap(); - /// - /// // Use a dynamic check to prove that it's okay to return concurrent - /// // references to `packet` and `rest`. - /// let (packet, rest) = split.via_runtime_check().unwrap(); - /// - /// assert_eq!(packet.length, 4); - /// assert_eq!(packet.body, [1, 2, 3, 4]); - /// assert_eq!(rest, [5, 6, 7, 8, 9]); - /// - /// rest.fill(0); + /// // The two-byte prefix and four-byte body need no trailing padding. + /// if let Some(split) = packet.split_at_mut(packet.length as usize) { + /// if let Ok((packet, rest)) = split.via_runtime_check() { + /// assert_eq!(packet.length, 4); + /// assert_eq!(packet.body, [1, 2, 3, 4]); + /// assert_eq!(rest, [5, 6, 7, 8, 9]); + /// rest.fill(0); + /// } else { + /// unreachable!("A four-byte body requires no trailing padding"); + /// } + /// } else { + /// unreachable!("The packet's length field is within its body"); /// } /// /// assert_eq!(packet.length, 4); @@ -805,7 +928,7 @@ where impl<'a, T, I> Split> where T: ?Sized + SplitAt, - I: Invariants, + I: Invariants, { fn into_ref(self) -> Split<&'a T> where @@ -860,37 +983,57 @@ where unsafe { self.via_unchecked() } } - /// Produces the split parts of `self`, using [`Unaligned`] to ensure that - /// it is sound to have concurrent references to both parts. + /// Produces the split parts of `self` after the conservative layout + /// predicate establishes that `T` never requires dynamic trailing padding. #[inline(always)] - fn via_unaligned(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) - where - T: Unaligned, - { - // SAFETY: By `T: SplitAt + Unaligned`, `T` is either a slice or a - // `repr(C)` or `repr(transparent)` slice DST that is well-aligned at - // any address and length. If `T` is a slice DST with alignment 1, - // `repr(C)` or `repr(transparent)` ensures that no padding is placed - // after the final element of the trailing slice. Consequently, `T` can - // be split into strictly non-overlapping parts any any index. + fn via_no_dynamic_padding(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) { + static_assert!( + T: ?Sized + KnownLayout => !T::LAYOUT.requires_dynamic_padding(), + "`Split::via_no_dynamic_padding` cannot be used with a type whose layout may require dynamic trailing padding; use `Split::via_runtime_check` instead" + ); + + // SAFETY: The assertion establishes that `requires_dynamic_padding()` + // is false. By that method's guarantee, every valid metadata—including + // `self.l_len()`, which is valid by `Split`'s invariant—describes an + // object with no trailing padding. By the postcondition of + // `PtrInner::split_at_unchecked`, the two returned referents are then + // contiguous and non-overlapping. Since the source already conforms to + // `I::Aliasing`, splitting it into disjoint byte ranges preserves that + // invariant: under `Exclusive`, neither result aliases or accesses the + // other's bytes; under `Shared`, interior mutation through one result + // cannot mutate bytes pointed to by the other. This satisfies Rust's + // reference aliasing rules [1]. + // + // [1] Per https://doc.rust-lang.org/1.93.1/std/ptr/index.html#pointer-to-reference-conversion: + // + // When creating a mutable reference, then while this reference + // exists, the memory it points to must not get accessed (read or + // written) through any other pointer or reference not derived from + // this reference. + // + // When creating a shared reference, then while this reference + // exists, the memory it points to must not get mutated (except inside + // `UnsafeCell`). unsafe { self.via_unchecked() } } /// Produces the split parts of `self`, using a dynamic check to ensure that /// it is sound to have concurrent references to both parts. You should /// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or - /// [`Self::via_unaligned`], which have no runtime cost. + /// [`Self::via_no_dynamic_padding`], which perform no runtime check. #[inline(always)] fn via_runtime_check(self) -> Result<(Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>), Self> { let l_len = self.l_len(); + // `l_len` witnesses an object size at most `isize::MAX`, and + // `KnownLayout::LAYOUT` guarantees that it contains the trailing + // slice. Thus `padding_for_elems` returns the exact trailing padding. + let trailing_padding = crate::trailing_slice_layout::().padding_for_elems(l_len.get()); // FIXME(#1290): Once we require `KnownLayout` on all fields, add an // `IS_IMMUTABLE` associated const, and add `T::IS_IMMUTABLE ||` to the // below check. - if l_len.padding_needed_for() == 0 { - // SAFETY: By `T: SplitAt`, `T` is either `[T]`, or a `repr(C)` or - // `repr(transparent)` slice DST, for which the trailing padding - // needed to accommodate `l_len` trailing elements is - // `l_len.padding_needed_for()`. If no trailing padding is required, + if trailing_padding == 0 { + // SAFETY: As established above, `trailing_padding` is the exact + // padding after the left part's trailing slice. If it is zero, // the left and right parts are strictly non-overlapping. Ok(unsafe { self.via_unchecked() }) } else { @@ -903,7 +1046,7 @@ where /// # Safety /// /// The caller promises that if `I::Aliasing` is [`Exclusive`] or `T` - /// permits interior mutation, then `l_len.padding_needed_for() == 0`. + /// permits interior mutation, then the left part has no trailing padding. #[inline(always)] unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) { let l_len = self.l_len(); @@ -915,11 +1058,11 @@ where // Lemma 0: `left` and `right` conform to the aliasing invariant // `I::Aliasing`. Proof: If `I::Aliasing` is `Exclusive` or `T` permits - // interior mutation, the caller promises that `l_len.padding_needed_for() - // == 0`. Consequently, by post-condition on `PtrInner::split_at_unchecked`, - // there is no trailing padding after `left`'s final element that would - // overlap into `right`. If `I::Aliasing` is shared and `T` forbids interior - // mutation, then overlap between their referents is permissible. + // interior mutation, the caller promises that the left part has no + // trailing padding. Consequently, by the postcondition of + // `PtrInner::split_at_unchecked`, `left` and `right` do not overlap. + // If `I::Aliasing` is shared and `T` forbids interior mutation, then + // overlap between their referents is permissible. // SAFETY: // 0. `left` conforms to the aliasing invariant of `I::Aliasing`, by Lemma 0. @@ -933,17 +1076,19 @@ where // SAFETY: // 0. `right` conforms to the aliasing invariant of `I::Aliasing`, by Lemma // 0. - // 1. `right` conforms to the alignment invariant of `I::Alignment, because - // if `ptr` with `I::Alignment = Aligned`, then by invariant on `T: - // SplitAt`, the trailing slice of `ptr` (from which `right` is derived) - // will also be well-aligned. + // 1. `right` conforms to the alignment invariant of `I::Alignment`: + // if the source is aligned, `T: SplitAt` guarantees that its + // trailing slice's address is aligned for `T::Elem`. Advancing by + // whole elements preserves that alignment. // 2. `right` conforms to the validity invariant of `I::Validity`, - // because `right: [T::Elem]` is derived from the trailing slice of - // `ptr`, which, by contract on `T: SplitAt::Elem`, has type - // `[T::Elem]`. The `left` part cannot be used to invalidate `right`, - // because the caller promises that if `I::Aliasing` is `Exclusive` - // or `T` permits interior mutation, then `l_len.padding_needed_for() - // == 0` and thus the parts will be non-overlapping. + // because `T: SplitAt` guarantees that `T::Elem` has the size and + // bit validity of the original trailing element. It also guarantees + // compatible shared and mutable access, so writes through `right` + // preserve the original elements' validity and interior mutation + // respects the original `UnsafeCell` coverage. The left part + // cannot invalidate `right`: when the references are exclusive or + // permit interior mutation, the caller guarantees no trailing + // padding and hence no overlap. let right = unsafe { Ptr::from_inner(right) }; (left, right) @@ -952,6 +1097,185 @@ where #[cfg(test)] mod tests { + use core::{cell::Cell, mem::ManuallyDrop}; + + use crate::{Immutable, KnownLayout, SplitAt, Unalign, Unaligned}; + + #[derive(KnownLayout, SplitAt, Immutable, Unaligned)] + #[repr(C, packed)] + struct Packed { + prefix: u8, + tail: ManuallyDrop, + } + + #[derive(KnownLayout, SplitAt, Immutable)] + #[repr(C, packed(2))] + struct Packed2 { + prefix: u8, + tail: ManuallyDrop, + } + + #[derive(KnownLayout, SplitAt, Immutable)] + #[repr(C)] + struct Inner { + prefix: u32, + tail: T, + } + + #[test] + fn test_split_at_packed() { + use crate::{FromBytes, IntoBytes}; + + #[derive(FromBytes, IntoBytes, KnownLayout, SplitAt, Immutable, Unaligned)] + #[repr(C, packed)] + struct Packet { + prefix: u8, + tail: [u16], + } + + // The tail starts at an odd address, exercising unaligned reads and + // writes on targets where `u16` has alignment 2. + #[repr(align(8))] + struct Bytes([u8; 9]); + + for i in 0..=4 { + let mut bytes = Bytes([1; 9]); + let packet = Packet::ref_from_bytes(&bytes.0).unwrap(); + let (left, right): (&Packet, &[Unalign]) = + packet.split_at(i).unwrap().via_immutable(); + assert_eq!(core::mem::size_of_val(left), 1 + 2 * i); + assert_eq!(right.len(), 4 - i); + assert!(right.iter().all(|elem| elem.get() == 0x0101)); + assert!(left.split_at(i).is_some()); + assert!(left.split_at(i + 1).is_none()); + assert!(packet.split_at(5).is_none()); + + let (_, right) = packet.split_at(i).unwrap().via_into_bytes(); + assert_eq!(right.len(), 4 - i); + let (_, right) = packet.split_at(i).unwrap().via_no_dynamic_padding(); + assert_eq!(right.len(), 4 - i); + + let packet = Packet::mut_from_bytes(&mut bytes.0).unwrap(); + let (left, right): (&mut Packet, &mut [Unalign]) = + packet.split_at_mut(i).unwrap().via_runtime_check().ok().unwrap(); + left.prefix = 2; + for elem in right { + *elem = Unalign::new(0x0202); + } + assert_eq!(bytes.0[0], 2); + assert!(bytes.0[1..1 + 2 * i].iter().all(|&byte| byte == 1)); + assert!(bytes.0[1 + 2 * i..].iter().all(|&byte| byte == 2)); + } + } + + #[test] + fn test_split_at_nested_packed() { + // A packing factor greater than 1 still exposes `Unalign` elements, + // including when the factor is below the original element alignment. + let mut words = Packed2 { prefix: 0, tail: ManuallyDrop::new([1u32, 2, 3]) }; + let dst: &mut Packed2<[u32]> = &mut words; + let (left, right): (&mut _, &mut [Unalign]) = + dst.split_at_mut(1).unwrap().via_no_dynamic_padding(); + left.prefix = 4; + right[0] = Unalign::new(5); + let dst: &Packed2<[u32]> = &words; + let (left, right) = dst.split_at(0).unwrap().via_immutable(); + assert_eq!(left.prefix, 4); + assert_eq!(right[0].get(), 1); + assert_eq!(right[1].get(), 5); + assert_eq!(right[2].get(), 3); + + let mut packet = Inner { + prefix: 0, + tail: Packed2 { + prefix: 1, + tail: ManuallyDrop::new(Packed { + prefix: 2, + tail: ManuallyDrop::new([3u16, 4, 5]), + }), + }, + }; + let dst: &Inner>> = &packet; + let (left, right): (&_, &[Unalign>]) = + dst.split_at(1).unwrap().via_immutable(); + assert_eq!(left.prefix, 0); + assert_eq!(right.len(), 2); + assert_eq!(right[0].get().get(), 4); + assert_eq!(right[1].get().get(), 5); + + // Split the packed field separately: the unpacked outer struct's + // padding does not constrain this split. + let dst: &mut Packed2> = &mut packet.tail; + let (left, right): (&mut _, &mut [Unalign>]) = + dst.split_at_mut(1).unwrap().via_no_dynamic_padding(); + left.prefix = 6; + right[0] = Unalign::new(Unalign::new(7)); + let dst: &Inner>> = &packet; + let (_, right) = dst.split_at(0).unwrap().via_immutable(); + assert_eq!(right[0].get().get(), 3); + assert_eq!(right[1].get().get(), 7); + assert_eq!(right[2].get().get(), 5); + } + + #[test] + fn test_split_at_packed_preserves_inner_padding() { + for i in 0..=4 { + let mut packet = Packed { + prefix: 1, + tail: ManuallyDrop::new(Inner { prefix: 0, tail: [2u8, 3, 4, 5] }), + }; + let dst: &Packed> = &packet; + let (left, right): (&_, &[Unalign]) = dst.split_at(i).unwrap().via_immutable(); + assert_eq!(right.len(), 4 - i); + assert_eq!(right.first().map(Unalign::get), [2, 3, 4, 5].get(i).copied()); + + // The inner `u32` prefix makes the tail start at byte 5 of the + // packed wrapper. Its size may still include inner padding. + let has_padding = core::mem::size_of_val(left) != 5 + i; + assert_eq!(dst.split_at(i).unwrap().via_runtime_check().is_err(), has_padding); + + let dst: &mut Packed> = &mut packet; + let split = dst.split_at_mut(i).unwrap().via_runtime_check(); + assert_eq!(split.is_err(), has_padding); + if let Ok((left, right)) = split { + left.prefix = 6; + for elem in right { + *elem = Unalign::new(7); + } + let dst: &Packed> = &packet; + let (left, right) = dst.split_at(i).unwrap().via_immutable(); + assert_eq!(left.prefix, 6); + assert!(right.iter().all(|elem| elem.get() == 7)); + } + } + } + + #[test] + fn test_split_at_packed_interior_mutation_and_zsts() { + let packet = Packed { + prefix: 0, + tail: ManuallyDrop::new([Cell::new(1u8), Cell::new(2), Cell::new(3)]), + }; + let dst: &Packed<[Cell]> = &packet; + let (left, right): (&_, &[Unalign>]) = + dst.split_at(1).unwrap().via_no_dynamic_padding(); + left.tail[0].set(4); + right[0].try_deref().unwrap().set(5); + assert_eq!(dst.tail[0].get(), 4); + assert_eq!(dst.tail[1].get(), 5); + assert_eq!(dst.tail[2].get(), 3); + + let mut packet = Packed { prefix: 0, tail: ManuallyDrop::new([(); 3]) }; + for i in 0..=3 { + let dst: &mut Packed<[()]> = &mut packet; + let (left, right): (&mut _, &mut [Unalign<()>]) = + dst.split_at_mut(i).unwrap().via_no_dynamic_padding(); + assert_eq!(left.tail.len(), i); + assert_eq!(right.len(), 3 - i); + assert_eq!(core::mem::size_of_val(left), 1); + } + } + #[cfg(feature = "derive")] #[test] fn test_split_at() { @@ -1011,6 +1335,8 @@ mod tests { trailing: [u8], } + assert!(SliceDst::LAYOUT.requires_dynamic_padding()); + const N: usize = 16; let arr = [1u16; N]; @@ -1070,21 +1396,79 @@ mod tests { assert_eq!(r, &[4]); } #[test] + #[allow(deprecated)] fn test_split_at_via_unaligned() { - use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt, Unaligned}; - #[derive(FromBytes, KnownLayout, SplitAt, IntoBytes, Immutable, Unaligned)] + use crate::{Immutable, KnownLayout, Split, SplitAt, Unaligned}; + + fn via_unaligned<'a, T>(split: Split<&'a T>) -> (&'a T, &'a [T::Elem]) + where + T: ?Sized + SplitAt + Unaligned, + { + split.via_unaligned() + } + + fn via_unaligned_mut<'a, T>(split: Split<&'a mut T>) -> (&'a mut T, &'a mut [T::Elem]) + where + T: ?Sized + SplitAt + Unaligned, + { + split.via_unaligned() + } + + #[derive(KnownLayout, SplitAt, Immutable, Unaligned)] #[repr(C)] - struct Packet { - length: u8, - body: [u8], + struct Packet { + prefix: [u8; 2], + body: B, } - let arr = [1, 2, 3, 4]; - let packet = Packet::ref_from_bytes(&arr[..]).unwrap(); + // Exercise generic callers using the original `T: Unaligned` bound. + assert!(!Packet::<[[u8; 2]]>::LAYOUT.requires_dynamic_padding()); + let packet = Packet { prefix: [0, 1], body: [[2, 3], [4, 5], [6, 7]] }; + let packet: &Packet<[[u8; 2]]> = &packet; let split = packet.split_at(2).unwrap(); - let (l, r) = split.via_unaligned(); - assert_eq!(l.length, 1); - assert_eq!(r, &[4]); + let (l, r) = via_unaligned(split); + assert_eq!(l.body, [[2, 3], [4, 5]]); + assert_eq!(r, &[[6, 7]]); + + let mut packet = Packet { prefix: [0, 1], body: [[2, 3], [4, 5], [6, 7]] }; + { + let packet: &mut Packet<[[u8; 2]]> = &mut packet; + let split = packet.split_at_mut(2).unwrap(); + let (l, r) = via_unaligned_mut(split); + l.body[0] = [8, 9]; + r[0] = [10, 11]; + } + assert_eq!(packet.body, [[8, 9], [4, 5], [10, 11]]); + } + + #[test] + fn test_split_at_via_no_dynamic_padding() { + use core::cell::Cell; + + use crate::SplitAt; + + // `u16` does not implement `Unaligned`, but `[u16]` never requires + // dynamic trailing padding. + let words = [1u16, 2, 3, 4]; + let split = SplitAt::split_at(&words[..], 2).unwrap(); + let (left, right) = split.via_no_dynamic_padding(); + assert_eq!(left, [1, 2]); + assert_eq!(right, [3, 4]); + + let mut words = [1u16, 2, 3, 4]; + let split = SplitAt::split_at_mut(&mut words[..], 2).unwrap(); + let (left, right) = split.via_no_dynamic_padding(); + left[0] = 5; + right[0] = 6; + assert_eq!(words, [5, 2, 6, 4]); + + // Exercise the safety-sensitive shared route with interior mutation. + let cells = [Cell::new(1u16), Cell::new(2), Cell::new(3)]; + let split = SplitAt::split_at(&cells[..], 2).unwrap(); + let (left, right) = split.via_no_dynamic_padding(); + left[0].set(4); + right[0].set(5); + assert_eq!([cells[0].get(), cells[1].get(), cells[2].get()], [4, 2, 5]); } } diff --git a/rust/zerocopy/src/util/macro_util.rs b/rust/zerocopy/src/util/macro_util.rs index ceeb80432b0ba8..fdcaa58d418cda 100644 --- a/rust/zerocopy/src/util/macro_util.rs +++ b/rust/zerocopy/src/util/macro_util.rs @@ -28,8 +28,8 @@ use core::{marker::PhantomData, mem, num::Wrapping}; use crate::{ pointer::{ - cast::CastSized, - invariant::{Aligned, Initialized, Valid}, + cast::CastSizedExact, + invariant::{Aligned, Initialized, Safe}, BecauseImmutable, }, FromBytes, Immutable, IntoBytes, KnownLayout, Ptr, ReadOnly, TryFromBytes, ValidityError, @@ -518,9 +518,9 @@ pub const fn hash_name(name: &str) -> i128 { /// /// `try_transmute` may either produce a post-monomorphization error or a panic /// if `Dst` is bigger than `Src`. Otherwise, `try_transmute` panics under the -/// same circumstances as [`is_bit_valid`]. +/// same circumstances as [`is_safe`]. /// -/// [`is_bit_valid`]: TryFromBytes::is_bit_valid +/// [`is_safe`]: TryFromBytes::is_safe #[inline(always)] pub fn try_transmute(src: Src) -> Result> where @@ -529,30 +529,64 @@ where { static_assert!(Src, Dst => mem::size_of::() == mem::size_of::()); - let mu_src = mem::MaybeUninit::new(src); - // SAFETY: `MaybeUninit` has no validity requirements. - let mu_dst: mem::MaybeUninit> = - unsafe { crate::util::transmute_unchecked(mu_src) }; - - let ptr = Ptr::from_ref(&mu_dst); - - // SAFETY: Since `Src: IntoBytes`, and since `size_of::() == - // size_of::()` by the preceding assertion, all of `mu_dst`'s bytes are - // initialized. `MaybeUninit` has no validity requirements, so even if - // `ptr` is used to mutate its referent (which it actually can't be - it's - // a shared `ReadOnly` pointer), that won't violate its referent's validity. - let ptr = unsafe { ptr.assume_validity::() }; - if Dst::is_bit_valid(ptr.cast::<_, CastSized, _>()) { - // SAFETY: Since `Dst::is_bit_valid`, we know that `ptr`'s referent is - // bit-valid for `Dst`. `ptr` points to `mu_dst`, and no intervening - // operations have mutated it, so it is a bit-valid `Dst`. - Ok(ReadOnly::into_inner(unsafe { mu_dst.assume_init() })) + let src = mem::ManuallyDrop::new(ReadOnly::new(src)); + let mut ptr = + Ptr::from_ref(&*src).transmute_with::, Initialized, CastSizedExact, _>(); + if Dst::is_safe(ptr.reborrow_shared()) { + // SAFETY: `read_unaligned` requires that `ptr` be valid for reads and + // point to a properly initialized `ReadOnly` [1]. `ptr` was + // derived from a live reference to `src`; `CastSizedExact` preserves + // that reference's address, provenance, and exact byte extent. Thus, + // the entire range remains live and valid for reads. The reference is + // non-null even when `ReadOnly` is zero-sized, and the cast + // preserves that address, satisfying [1]'s explicit ZST requirement. + // + // `Src: IntoBytes` and the `ReadOnly` bridge let `transmute_with` + // establish that every byte in that exact range is initialized. + // `Dst::is_safe` then established that the same bytes are bit-valid for + // `Dst`, and thus for `ReadOnly`. The candidate was a shared + // `ReadOnly` pointer, so it could not mutate the bytes; no operation + // occurs between validation and this read which could do so. Alignment + // is not an additional precondition because [1] expressly permits + // unaligned pointers. + // + // `read_unaligned` leaves the source memory unchanged and makes a + // bitwise copy [1]. The source is wrapped in `ManuallyDrop`, which + // inhibits its destructor [2]. After this read, neither `src` nor + // `ptr` is used to access the inner value, and the wrapper's eventual + // drop does not drop it. Thus, only the returned `Dst` is subsequently + // used or dropped; there is no duplicate ownership use. + // + // [1] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.read_unaligned.html: + // + // Reads the value from `src` without moving it. This leaves the + // memory in `src` unchanged. + // + // Unlike `read`, `read_unaligned` works with unaligned pointers. + // + // Behavior is undefined if any of the following conditions are + // violated: + // + // * `src` must be valid for reads. + // + // * `src` must point to a properly initialized value of type `T`. + // + // Like `read`, `read_unaligned` creates a bitwise copy of `T`, + // regardless of whether `T` is `Copy`. If `T` is not `Copy`, using + // both the returned value and the value at `*src` can violate + // memory safety. + // + // Note that even if `T` has size `0`, the pointer must be non-null. + // + // [2] Per https://doc.rust-lang.org/1.56.0/std/mem/struct.ManuallyDrop.html: + // + // A wrapper to inhibit compiler from automatically calling `T`'s + // destructor. + let dst = unsafe { core::ptr::read_unaligned(ptr.as_inner().as_ptr()) }; + Ok(ReadOnly::into_inner(dst)) } else { - // SAFETY: `MaybeUninit` has no validity requirements. - let mu_src: mem::MaybeUninit = unsafe { crate::util::transmute_unchecked(mu_dst) }; - // SAFETY: `mu_dst`/`mu_src` was constructed from `src` and never - // modified, so it is still bit-valid. - Err(ValidityError::new(unsafe { mu_src.assume_init() })) + let src = ReadOnly::into_inner(mem::ManuallyDrop::into_inner(src)); + Err(ValidityError::new(src)) } } @@ -599,7 +633,7 @@ where let res = ptr.try_with(#[inline(always)] |ptr| { let ptr = ptr.recall_validity::(); let ptr = ptr.cast::<_, crate::layout::CastFrom, _>(); - ptr.try_into_valid() + ptr.try_into_safe() }); match res { Ok(ptr) => { @@ -662,7 +696,7 @@ where ptr.try_with_unchecked(#[inline(always)] |ptr| { let ptr = ptr.recall_validity::(); let ptr = ptr.cast::<_, crate::layout::CastFrom, _>(); - ptr.try_into_valid() + ptr.try_into_safe() }) }; match res { @@ -878,7 +912,7 @@ where let ptr = Ptr::from_ref(self.0) .recall_validity::() .transmute_with::, (crate::pointer::BecauseMutationCompatible, _)>() - .recall_validity::(); + .recall_validity::(); static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => { Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get() @@ -911,7 +945,7 @@ where let ptr = Ptr::from_mut(self.0) .recall_validity::() .transmute_with::, _>() - .recall_validity::(); + .recall_validity::(); static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => { Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get() @@ -946,6 +980,22 @@ pub mod core_reexport { } } +/// Projects `Self` to itself. +/// +/// This is used by macros to have the compiler verify that two syntactically +/// similar types actually resolve to the same type. The blanket implementation +/// ensures that no other implementation can project a type to anything other +/// than itself. +#[doc(hidden)] +pub trait Identity { + /// `Self`. + type Type: ?Sized; +} + +impl Identity for T { + type Type = T; +} + #[cfg(test)] mod tests { use core::num::NonZeroUsize; diff --git a/rust/zerocopy/src/util/macros.rs b/rust/zerocopy/src/util/macros.rs index 7e63e3a54fc41e..e514407ceeaa07 100644 --- a/rust/zerocopy/src/util/macros.rs +++ b/rust/zerocopy/src/util/macros.rs @@ -15,21 +15,20 @@ /// The trait impl must be sound. /// /// When implementing `TryFromBytes`: -/// - If no `is_bit_valid` impl is provided, then it must be valid for -/// `is_bit_valid` to unconditionally return `true`. In other words, it must -/// be the case that any initialized sequence of bytes constitutes a valid -/// instance of `$ty`. -/// - If an `is_bit_valid` impl is provided, then the impl of `is_bit_valid` -/// must only return `true` if its argument refers to a valid `$ty`. +/// - If no `is_safe` impl is provided, then it must be valid for `is_safe` to +/// unconditionally return `true`. In other words, it must be the case that any +/// initialized sequence of bytes constitutes a valid instance of `$ty`. +/// - If an `is_safe` impl is provided, then the impl of `is_safe` must only +/// return `true` if its argument refers to a valid `$ty`. macro_rules! unsafe_impl { // Implement `$trait` for `$ty` with no bounds. - ($(#[$attr:meta])* $ty:ty: $trait:ident $(; |$candidate:ident| $is_bit_valid:expr)?) => {{ + ($(#[$attr:meta])* $ty:ty: $trait:ident $(; |$candidate:ident| $is_safe:expr)?) => {{ crate::util::macros::__unsafe(); $(#[$attr])* // SAFETY: The caller promises that this is sound. unsafe impl $trait for $ty { - unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?); + unsafe_impl!(@method $trait $(; |$candidate| $is_safe)?); } }}; @@ -92,26 +91,26 @@ macro_rules! unsafe_impl { $(#[$attr:meta])* const $constname:ident : $constty:ident $(,)? $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),* - => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)? + => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)? ) => { unsafe_impl!( @inner $(#[$attr])* @const $constname: $constty, $($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)* - => $trait for $ty $(; |$candidate| $is_bit_valid)? + => $trait for $ty $(; |$candidate| $is_safe)? ); }; ( $(#[$attr:meta])* $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),* - => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)? + => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)? ) => {{ unsafe_impl!( @inner $(#[$attr])* $($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)* - => $trait for $ty $(; |$candidate| $is_bit_valid)? + => $trait for $ty $(; |$candidate| $is_safe)? ); }}; ( @@ -119,7 +118,7 @@ macro_rules! unsafe_impl { $(#[$attr:meta])* $(@const $constname:ident : $constty:ident,)* $($tyvar:ident $(: $(? $optbound:ident +)* + $($bound:ident +)* )?,)* - => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)? + => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)? ) => {{ crate::util::macros::__unsafe(); @@ -127,21 +126,21 @@ macro_rules! unsafe_impl { #[allow(non_local_definitions)] // SAFETY: The caller promises that this is sound. unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),* $(, const $constname: $constty,)*> $trait for $ty { - unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?); + unsafe_impl!(@method $trait $(; |$candidate| $is_safe)?); } }}; - (@method TryFromBytes ; |$candidate:ident| $is_bit_valid:expr) => { + (@method TryFromBytes ; |$candidate:ident| $is_safe:expr) => { #[allow(clippy::missing_inline_in_public_items, dead_code)] #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))] fn only_derive_is_allowed_to_implement_this_trait() {} #[inline] - fn is_bit_valid($candidate: Maybe<'_, Self, Alignment>) -> bool + fn is_safe($candidate: Maybe<'_, Self, Alignment>) -> bool where Alignment: crate::invariant::Alignment, { - $is_bit_valid + $is_safe } }; (@method TryFromBytes) => { @@ -149,7 +148,7 @@ macro_rules! unsafe_impl { #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))] fn only_derive_is_allowed_to_implement_this_trait() {} #[inline(always)] - fn is_bit_valid(_candidate: Maybe<'_, Self, Alignment>) -> bool + fn is_safe(_candidate: Maybe<'_, Self, Alignment>) -> bool where Alignment: crate::invariant::Alignment, { @@ -161,16 +160,16 @@ macro_rules! unsafe_impl { #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))] fn only_derive_is_allowed_to_implement_this_trait() {} }; - (@method $trait:ident; |$_candidate:ident| $_is_bit_valid:expr) => { - compile_error!("Can't provide `is_bit_valid` impl for trait other than `TryFromBytes`"); + (@method $trait:ident; |$_candidate:ident| $_is_safe:expr) => { + compile_error!("Can't provide `is_safe` impl for trait other than `TryFromBytes`"); }; } /// Implements `$trait` for `$ty` where `$ty: TransmuteFrom<$repr>` (and /// vice-versa). /// -/// Calling this macro is safe; the internals of the macro emit appropriate -/// trait bounds which ensure that the given impl is sound. +/// This macro is intended to be safe to call. The current proof is incomplete; +/// see FIXME(#3691) below. macro_rules! impl_for_transmute_from { ( $(#[$attr:meta])* @@ -181,26 +180,32 @@ macro_rules! impl_for_transmute_from { $(#[$attr])* #[allow(non_local_definitions)] - // SAFETY: `is_trait` (defined and used below) requires `T: - // TransmuteFrom`, `R: TransmuteFrom`, and `R: $trait`. It is - // called using `$ty` and `$repr`, ensuring that `$ty` and `$repr` - // have equivalent bit validity, and ensuring that `$repr: $trait`. - // The supported traits - `TryFromBytes`, `FromZeros`, `FromBytes`, - // and `IntoBytes` - are defined only in terms of the bit validity - // of a type. Therefore, `$repr: $trait` ensures that `$ty: $trait` - // is sound. + // SAFETY: `is_trait` (defined and used below) requires + // reciprocal `TransmuteFrom<_, Safe, Safe>` bounds and `R: $trait`. + // If `T` and `R` have the same size, the reciprocal bounds imply + // that they permit the same `Safe` bit patterns. The call below + // instantiates `T` with `$ty` and `R` with `$repr`, and establishes + // `$repr: $trait`. The supported traits - `TryFromBytes`, + // `FromZeros`, `FromBytes`, and `IntoBytes` - are defined only in + // terms of bit validity, so these premises are sufficient when + // `$ty` and `$repr` have the same size. + // + // FIXME(#3691): This macro does not establish that `$ty` and + // `$repr` have the same size. Without that premise, `TransmuteFrom` + // conveys no safety guarantee, so these bounds do not make + // arbitrary invocations of this macro sound. unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?> $trait for $ty { #[allow(dead_code, clippy::missing_inline_in_public_items)] #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))] fn only_derive_is_allowed_to_implement_this_trait() { - use crate::pointer::{*, invariant::Valid}; + use crate::pointer::{*, invariant::Safe}; impl_for_transmute_from!(@assert_is_supported_trait $trait); fn is_trait() where - T: TransmuteFrom + ?Sized, - R: TransmuteFrom + ?Sized, + T: TransmuteFrom + ?Sized, + R: TransmuteFrom + ?Sized, R: $trait, { } @@ -212,7 +217,7 @@ macro_rules! impl_for_transmute_from { } impl_for_transmute_from!( - @is_bit_valid + @is_safe $(<$tyvar $(: $(? $optbound +)* $($bound +)*)?>)? $trait for $ty [$repr] ); @@ -224,27 +229,27 @@ macro_rules! impl_for_transmute_from { (@assert_is_supported_trait FromBytes) => {}; (@assert_is_supported_trait IntoBytes) => {}; ( - @is_bit_valid + @is_safe $(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)? TryFromBytes for $ty:ty [$repr:ty] ) => { #[inline(always)] - fn is_bit_valid(candidate: $crate::Maybe<'_, Self, Alignment>) -> bool + fn is_safe(candidate: $crate::Maybe<'_, Self, Alignment>) -> bool where Alignment: $crate::invariant::Alignment, { // SAFETY: This macro ensures that `$repr` and `Self` have the same // size and bit validity. Thus, a bit-valid instance of `$repr` is // also a bit-valid instance of `Self`. - <$repr as TryFromBytes>::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>()) + <$repr as TryFromBytes>::is_safe(candidate.transmute::<_, _, BecauseImmutable>()) } }; ( - @is_bit_valid + @is_safe $(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)? $trait:ident for $ty:ty [$repr:ty] ) => { - // Trait other than `TryFromBytes`; no `is_bit_valid` impl. + // Trait other than `TryFromBytes`; no `is_safe` impl. }; } @@ -269,33 +274,33 @@ macro_rules! impl_for_transmute_from { macro_rules! unsafe_impl_for_power_set { ( $first:ident $(, $rest:ident)* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...) - $(; |$candidate:ident| $is_bit_valid:expr)? + $(; |$candidate:ident| $is_safe:expr)? ) => { unsafe_impl_for_power_set!( $($rest),* $(-> $ret)? => $trait for $macro!(...) - $(; |$candidate| $is_bit_valid)? + $(; |$candidate| $is_safe)? ); unsafe_impl_for_power_set!( @impl $first $(, $rest)* $(-> $ret)? => $trait for $macro!(...) - $(; |$candidate| $is_bit_valid)? + $(; |$candidate| $is_safe)? ); }; ( $(-> $ret:ident)? => $trait:ident for $macro:ident!(...) - $(; |$candidate:ident| $is_bit_valid:expr)? + $(; |$candidate:ident| $is_safe:expr)? ) => { unsafe_impl_for_power_set!( @impl $(-> $ret)? => $trait for $macro!(...) - $(; |$candidate| $is_bit_valid)? + $(; |$candidate| $is_safe)? ); }; ( @impl $($vars:ident),* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...) - $(; |$candidate:ident| $is_bit_valid:expr)? + $(; |$candidate:ident| $is_safe:expr)? ) => { unsafe_impl!( $($vars,)* $($ret)? => $trait for $macro!($($vars),* $(-> $ret)?) - $(; |$candidate| $is_bit_valid)? + $(; |$candidate| $is_safe)? ); }; } @@ -355,8 +360,8 @@ macro_rules! opt_unsafe_fn { /// impl in this case, and also provides useful documentation for readers of the /// code. /// -/// Finally, if a `TryFromBytes::is_bit_valid` impl is provided, it must adhere -/// to the safety preconditions of [`unsafe_impl!`]. +/// Finally, if a `TryFromBytes::is_safe` impl is provided, it must adhere to the +/// safety preconditions of [`unsafe_impl!`]. /// /// ## Example /// @@ -396,11 +401,11 @@ macro_rules! impl_or_verify { }; ( $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),* - => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)? + => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)? ) => { impl_or_verify!(@impl { unsafe_impl!( $($tyvar $(: $(? $optbound +)* $($bound +)*)?),* => $trait for $ty - $(; |$candidate| $is_bit_valid)? + $(; |$candidate| $is_safe)? ); }); impl_or_verify!(@verify $trait, { impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {} @@ -495,7 +500,7 @@ macro_rules! unsafe_impl_known_layout { #[allow(non_local_definitions)] // SAFETY: The caller promises that this is sound. unsafe impl<$($tyvar: ?Sized + KnownLayout)?> KnownLayout for $ty { - #[allow(clippy::missing_inline_in_public_items, dead_code)] + #[allow(dead_code, clippy::missing_inline_in_public_items)] #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))] fn only_derive_is_allowed_to_implement_this_trait() {} @@ -742,14 +747,14 @@ macro_rules! unsafe_impl_for_transparent_wrapper { ($vis:vis T $(: ?$optbound:ident)? => $wrapper:ident) => {{ crate::util::macros::__unsafe(); - use crate::pointer::{TransmuteFrom, cast::{CastExact, TransitiveProject}, SizeEq, invariant::Valid}; + use crate::pointer::{TransmuteFrom, cast::{CastExact, TransitiveProject}, SizeEq, invariant::Safe}; use crate::wrappers::ReadOnly; // SAFETY: The caller promises that `T` and `$wrapper` have the same // bit validity. - unsafe impl TransmuteFrom for $wrapper {} + unsafe impl TransmuteFrom for $wrapper {} // SAFETY: See previous safety comment. - unsafe impl TransmuteFrom<$wrapper, Valid, Valid> for T {} + unsafe impl TransmuteFrom<$wrapper, Safe, Safe> for T {} // SAFETY: The caller promises that a `T` to `$wrapper` cast is // size-preserving. define_cast!(unsafe { $vis CastToWrapper = T => $wrapper }); @@ -805,7 +810,7 @@ macro_rules! unsafe_impl_for_transparent_wrapper { macro_rules! impl_transitive_transmute_from { ($($tyvar:ident $(: ?$optbound:ident)?)? => $t:ty => $u:ty => $v:ty) => { const _: () = { - use crate::pointer::{TransmuteFrom, SizeEq, invariant::Valid}; + use crate::pointer::{TransmuteFrom, SizeEq, invariant::Safe}; impl<$($tyvar $(: ?$optbound)?)?> SizeEq<$t> for $v where @@ -819,14 +824,14 @@ macro_rules! impl_transitive_transmute_from { >; } - // SAFETY: Since `$u: TransmuteFrom<$t, Valid, Valid>`, it is sound - // to transmute a bit-valid `$t` to a bit-valid `$u`. Since `$v: - // TransmuteFrom<$u, Valid, Valid>`, it is sound to transmute that + // SAFETY: Since `$u: TransmuteFrom<$t, Safe, Safe>`, it is sound to + // transmute a bit-valid `$t` to a bit-valid `$u`. Since `$v: + // TransmuteFrom<$u, Safe, Safe>`, it is sound to transmute that // bit-valid `$u` to a bit-valid `$v`. - unsafe impl<$($tyvar $(: ?$optbound)?)?> TransmuteFrom<$t, Valid, Valid> for $v + unsafe impl<$($tyvar $(: ?$optbound)?)?> TransmuteFrom<$t, Safe, Safe> for $v where - $u: TransmuteFrom<$t, Valid, Valid>, - $v: TransmuteFrom<$u, Valid, Valid>, + $u: TransmuteFrom<$t, Safe, Safe>, + $v: TransmuteFrom<$u, Safe, Safe>, {} }; }; diff --git a/rust/zerocopy/src/util/mod.rs b/rust/zerocopy/src/util/mod.rs index 02fd4ed62741b5..646f4573980205 100644 --- a/rust/zerocopy/src/util/mod.rs +++ b/rust/zerocopy/src/util/mod.rs @@ -23,7 +23,7 @@ use core::{ use super::*; use crate::pointer::{ - invariant::{Exclusive, Shared, Valid}, + invariant::{Exclusive, Safe, Shared}, SizeEq, TransmuteFromPtr, }; @@ -108,7 +108,7 @@ impl AsAddress for *const T { #[allow(clippy::as_conversions)] #[cfg_attr( __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, - allow(lossy_provenance_casts) + allow(implicit_provenance_casts) )] return self.cast::<()>() as usize; } @@ -144,9 +144,10 @@ pub(crate) fn validate_aligned_to(t: T) -> Result<(), Alignment /// on the answer it gives if this is not the case. #[cfg_attr( kani, - kani::requires(len <= DstLayout::MAX_SIZE), kani::requires(align.is_power_of_two()), - kani::ensures(|&p| (len + p) % align.get() == 0), + // A power-of-two alignment divides the `usize` modulus, so wrapping + // preserves congruence even when the next aligned value exceeds `usize`. + kani::ensures(|&p| len.wrapping_add(p) % align.get() == 0), // Ensures that we add the minimum required padding. kani::ensures(|&p| p < align.get()), )] @@ -340,7 +341,7 @@ pub(crate) unsafe fn transmute_ref(src: &Src) -> &Dst where Src: ?Sized, Dst: SizeEq - + TransmuteFromPtr>::CastFrom, R> + + TransmuteFromPtr>::CastFrom, R> + ?Sized, { let dst = Ptr::from_ref(src).transmute(); @@ -357,7 +358,7 @@ pub(crate) unsafe fn transmute_mut(src: &mut Src) -> &mut Dst where Src: ?Sized, Dst: SizeEq - + TransmuteFromPtr>::CastFrom, R> + + TransmuteFromPtr>::CastFrom, R> + ?Sized, { let dst = Ptr::from_mut(src).transmute(); @@ -562,31 +563,6 @@ mod len_of { self.meta } - #[inline] - pub(crate) fn padding_needed_for(&self) -> usize - where - T: KnownLayout, - { - let trailing_slice_layout = crate::trailing_slice_layout::(); - - // FIXME(#67): Remove this allow. See NumExt for more details. - #[allow( - unstable_name_collisions, - clippy::incompatible_msrv, - clippy::multiple_unsafe_ops_per_block - )] - // SAFETY: By invariant on `self`, a `&T` with metadata `self.meta` - // describes an object of size `<= isize::MAX`. This computes the - // size of such a `&T` without any trailing padding, and so neither - // the multiplication nor the addition will overflow. - let unpadded_size = unsafe { - let trailing_size = self.meta.unchecked_mul(trailing_slice_layout.elem_size); - trailing_size.unchecked_add(trailing_slice_layout.offset) - }; - - util::padding_needed_for(unpadded_size, T::LAYOUT.align) - } - #[inline(always)] pub(crate) fn validate_cast_and_convert_metadata( addr: usize, @@ -895,6 +871,8 @@ mod tests { #[test] fn test_round_down_to_next_multiple_of_alignment() { + // Division by `NonZeroUsize` unsupported on our MSRV. + #[allow(clippy::needless_nonzero_get)] fn alt_impl(n: usize, align: NonZeroUsize) -> usize { let mul = n / align.get(); mul * align.get() diff --git a/rust/zerocopy/src/wrappers.rs b/rust/zerocopy/src/wrappers.rs index 1a8cf2b41d5553..d0baa901f10c76 100644 --- a/rust/zerocopy/src/wrappers.rs +++ b/rust/zerocopy/src/wrappers.rs @@ -8,10 +8,10 @@ // This file may not be copied, modified, or distributed except according to // those terms. -use core::{fmt, hash::Hash}; +use core::{borrow::Borrow, fmt, hash::Hash}; use super::*; -use crate::pointer::{invariant::Valid, SizeEq, TransmuteFrom}; +use crate::pointer::{invariant::Safe, SizeEq, TransmuteFrom}; /// A type with no alignment requirement. /// @@ -140,10 +140,10 @@ impl_known_layout!(T => Unalign); // Unaligned`. // - `Unalign` has the same bit validity as `T`, and so it is `FromZeros`, // `FromBytes`, or `IntoBytes` exactly when `T` is as well. -// - `Immutable`: `Unalign` has the same fields as `T`, so it permits -// interior mutation exactly when `T` does. +// - `Immutable`: `Unalign` has the same fields as `T`, so it permits interior +// mutation exactly when `T` does. // - `TryFromBytes`: `Unalign` has the same the same bit validity as `T`, so -// `T::is_bit_valid` is a sound implementation of `is_bit_valid`. +// `T::is_safe` is a sound implementation of `is_safe`. // #[allow(clippy::multiple_unsafe_ops_per_block)] const _: () = unsafe { @@ -151,7 +151,7 @@ const _: () = unsafe { impl_or_verify!(T: Immutable => Immutable for Unalign); impl_or_verify!( T: TryFromBytes => TryFromBytes for Unalign; - |c| T::is_bit_valid(c.transmute::<_, _, BecauseImmutable>()) + |c| T::is_safe(c.transmute::<_, _, BecauseImmutable>()) ); impl_or_verify!(T: FromZeros => FromZeros for Unalign); impl_or_verify!(T: FromBytes => FromBytes for Unalign); @@ -613,7 +613,14 @@ mod read_only_def { /// Note that `&mut ReadOnly` still permits mutation – the read-only /// property only applies to shared references. /// + /// `ReadOnly` implements [`Copy`] and [`Clone`] when `T: Copy`. Cloning + /// copies the value without calling `T::clone`. Trait implementations that + /// expose or operate on a shared reference to `T` require `T: Immutable`. + /// Constructing a wrapper or accessing its contents through [`AsMut`] + /// does not require `T: Immutable`. + /// /// [`Immutable`]: crate::Immutable + #[derive(Copy)] #[repr(transparent)] pub struct ReadOnly { // INVARIANT: `inner` is never mutated through a `&ReadOnly` @@ -665,7 +672,7 @@ const _: () = unsafe { // SAFETY: // - `ReadOnly` has the same alignment as `T`, and so it is `Unaligned` // exactly when `T` is as well. -// - `ReadOnly` has the same bit validity as `T`, and so this `is_bit_valid` +// - `ReadOnly` has the same bit validity as `T`, and so this `is_safe` // implementation is correct, and thus the `TryFromBytes` impl is sound. // - `ReadOnly` has the same bit validity as `T`, and so it is `FromZeros`, // `FromBytes`, and `IntoBytes` exactly when `T` is as well. @@ -673,7 +680,7 @@ const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for ReadOnly); unsafe_impl!( T: ?Sized + TryFromBytes => TryFromBytes for ReadOnly; - |c| T::is_bit_valid(c.cast::<_, as SizeEq>>>::CastFrom, _>()) + |c| T::is_safe(c.cast::<_, as SizeEq>>>::CastFrom, _>()) ); unsafe_impl!(T: ?Sized + FromZeros => FromZeros for ReadOnly); unsafe_impl!(T: ?Sized + FromBytes => FromBytes for ReadOnly); @@ -709,11 +716,34 @@ const _: () = { // SAFETY: `ReadOnly` is a `#[repr(transparent)]` wrapper around `T`, and so // it has the same bit validity as `T`. -unsafe impl TransmuteFrom for ReadOnly {} +unsafe impl TransmuteFrom for ReadOnly {} // SAFETY: `ReadOnly` is a `#[repr(transparent)]` wrapper around `T`, and so // it has the same bit validity as `T`. -unsafe impl TransmuteFrom, Valid, Valid> for T {} +unsafe impl TransmuteFrom, Safe, Safe> for T {} + +impl Default for ReadOnly { + #[inline(always)] + fn default() -> Self { + Self::new(Default::default()) + } +} + +// Copying avoids calling `T::clone`, which could mutate through a shared +// reference to `T` and violate `ReadOnly`'s invariant. +impl Clone for ReadOnly { + #[inline(always)] + fn clone(&self) -> Self { + *self + } +} + +impl From for ReadOnly { + #[inline(always)] + fn from(t: T) -> Self { + Self::new(t) + } +} impl<'a, T: ?Sized + Immutable> From<&'a T> for &'a ReadOnly { #[inline(always)] @@ -743,15 +773,74 @@ impl DerefMut for ReadOnly { } } -impl Debug for ReadOnly { +impl AsRef for ReadOnly { #[inline(always)] - fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { - self.deref().fmt(f) + fn as_ref(&self) -> &T { + self.deref() + } +} + +impl Borrow for ReadOnly { + #[inline(always)] + fn borrow(&self) -> &T { + self.deref() + } +} + +impl AsMut for ReadOnly { + #[inline(always)] + fn as_mut(&mut self) -> &mut T { + ReadOnly::as_mut(self) + } +} + +// Delegate shared access through `Deref`, which requires `T: Immutable` and +// thus prevents interior mutation of the wrapped value by these trait methods. +impl PartialEq for ReadOnly { + #[inline(always)] + fn eq(&self, other: &Self) -> bool { + self.deref().eq(other.deref()) + } +} + +impl Eq for ReadOnly {} + +impl PartialOrd for ReadOnly { + #[inline(always)] + fn partial_cmp(&self, other: &Self) -> Option { + self.deref().partial_cmp(other.deref()) } } +impl Ord for ReadOnly { + #[inline(always)] + fn cmp(&self, other: &Self) -> Ordering { + self.deref().cmp(other.deref()) + } +} + +impl Hash for ReadOnly { + #[inline(always)] + fn hash(&self, state: &mut H) { + self.deref().hash(state); + } +} + +macro_rules! impl_read_only_fmt { + ($($trait:ident),* $(,)?) => {$( + impl fmt::$trait for ReadOnly { + #[inline(always)] + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + fmt::$trait::fmt(self.deref(), f) + } + } + )*}; +} + +impl_read_only_fmt!(Debug, Display, Binary, Octal, LowerHex, UpperHex, LowerExp, UpperExp); + // SAFETY: See safety comment on `ProjectToTag`. -unsafe impl HasTag for ReadOnly { +unsafe impl + ?Sized, Client> HasTag for ReadOnly { #[allow(clippy::missing_inline_in_public_items)] fn only_derive_is_allowed_to_implement_this_trait() where @@ -759,30 +848,33 @@ unsafe impl HasTag for ReadOnly { { } - type Tag = T::Tag; + type Tag = >::Tag; // SAFETY: `>>::CastFrom` is a no-op projection that - // produces a pointer with the same referent. By invariant, for any `Ptr<'_, - // T, I>` it is sound to use `T::ProjectToTag` to project to a `Ptr<'_, - // T::Tag, I>`. Since `ReadOnly` has the same layout and validity as `T`, - // the same is true of projecting from a `Ptr<'_, ReadOnly, I>`. + // produces a pointer with the same referent. By invariant, for any + // `Ptr<'_, T, I>` it is sound to use + // `>::ProjectToTag` to project to a + // shared `Ptr<'_, >::Tag, + // (_, I::Alignment, I::Validity)>`. Since `ReadOnly` has the same + // layout and validity as `T`, the same is true of projecting from a + // `Ptr<'_, ReadOnly, I>`. type ProjectToTag = crate::pointer::cast::TransitiveProject< T, >>::CastFrom, - T::ProjectToTag, + >::ProjectToTag, >; } // SAFETY: `ReadOnly` is a `#[repr(transparent)]` wrapper around `T`, and so // has the same fields at the same offsets. Thus, it satisfies the safety -// invariants of `HasField` for field `f` exactly -// when `T` does, as guaranteed by the `T: HasField` bound: -// - If `VARIANT_ID` is `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID`, then `T` has -// the layout of a struct or union type. Since `ReadOnly` is a transparent -// wrapper around `T`, it does too. Otherwise, if `VARIANT_ID` is an enum -// variant index, then `T` has the layout of an enum type, and `ReadOnly` -// does too. -// - By `T: HasField<_, _, FIELD_ID>`: +// invariants of `HasField` for field `f` +// exactly when `T` does, as guaranteed by the `T: HasField` bound: +// - If `VARIANT_ID` is `STRUCT_VARIANT_ID`, `UNION_VARIANT_ID`, or +// `REPR_C_UNION_VARIANT_ID`, then `T` has the layout of a struct, non-C union, +// or C union type respectively. Since `ReadOnly` is a transparent wrapper +// around `T`, it does too. Otherwise, `VARIANT_ID` is the identifier ID of an +// enum variant; `T` has the layout of an enum type, and `ReadOnly` does too. +// - By `T: HasField<_, _, _, FIELD_ID>`: // - `T` has a field `f` with name `n` such that // `FIELD_ID = zerocopy::ident_id!(n)` or at index `i` such that // `FIELD_ID = zerocopy::ident_id!(i)`. @@ -794,10 +886,10 @@ unsafe impl HasTag for ReadOnly { // refers to a non-strict subset of the bytes of `slf`'s referent, and has the // same provenance as `slf` – because all intermediate operations satisfy those // same conditions. -unsafe impl - HasField for ReadOnly +unsafe impl + HasField for ReadOnly where - T: HasField + ?Sized, + T: HasField + ?Sized, { #[allow(clippy::missing_inline_in_public_items)] fn only_derive_is_allowed_to_implement_this_trait() @@ -811,7 +903,7 @@ where #[inline(always)] fn project(slf: PtrInner<'_, Self>) -> *mut ReadOnly { slf.project::<_, >>::CastFrom>() - .project::<_, crate::pointer::cast::Projection>() + .project::<_, crate::pointer::cast::Projection>() .project::<_, as SizeEq>::CastFrom>() .as_non_null() .as_ptr() @@ -822,10 +914,10 @@ where // has the same fields at the same offsets. `is_projectable` simply delegates to // `T::is_projectable`, which is sound because a `Ptr<'_, ReadOnly, I>` will // be projectable exactly when a `Ptr<'_, T, I>` referent is. -unsafe impl - ProjectField for ReadOnly +unsafe impl + ProjectField for ReadOnly where - T: ProjectField + ?Sized, + T: ProjectField + ?Sized, I: invariant::Invariants, { #[allow(clippy::missing_inline_in_public_items)] @@ -835,13 +927,15 @@ where { } - type Invariants = T::Invariants; + type Invariants = >::Invariants; - type Error = T::Error; + type Error = >::Error; #[inline(always)] - fn is_projectable<'a>(ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> { - T::is_projectable(ptr) + fn is_projectable<'a>( + ptr: Ptr<'a, >::Tag, (invariant::Shared, I::Alignment, I::Validity)>, + ) -> Result<(), Self::Error> { + >::is_projectable(ptr) } } @@ -852,6 +946,96 @@ mod tests { use super::*; use crate::util::testutil::*; + #[test] + #[allow(clippy::clone_on_copy, clippy::non_canonical_clone_impl)] + fn test_read_only_copy_clone() { + // This type deliberately does not implement `Immutable`. Copying the + // wrapper must not require it or invoke the inner `Clone` impl. + #[derive(Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] + struct PanickingClone(u16); + + impl Clone for PanickingClone { + fn clone(&self) -> Self { + panic!("ReadOnly must not call the inner Clone implementation") + } + } + + static_assertions::assert_not_impl_any!( + ReadOnly: Debug, PartialEq, Eq, PartialOrd, Ord, + Hash, AsRef, Borrow + ); + static_assertions::assert_not_impl_any!( + ReadOnly>: Copy, Clone, Deref, AsRef>, Borrow> + ); + + let value = ReadOnly::new(PanickingClone(42)); + let copy = value; + let clone = value.clone(); + assert_eq!(ReadOnly::into_inner(copy).0, 42); + assert_eq!(ReadOnly::into_inner(clone).0, 42); + } + + #[test] + fn test_read_only_mutable_access() { + // Construction and exclusive access also support interior mutability. + let mut value = ReadOnly::>::default(); + let inner: &mut Cell = value.as_mut(); + inner.set(42); + assert_eq!(ReadOnly::into_inner(value).get(), 42); + + let value = ReadOnly::from(Cell::new(7)); + assert_eq!(ReadOnly::into_inner(value).get(), 7); + } + + #[test] + fn test_read_only_borrowed_lookup() { + // Borrowed lookups require equality, hashing, and ordering to agree + // between the wrapper and the inner value. + let mut hash_map = std::collections::HashMap::new(); + hash_map.insert(ReadOnly::new(42u16), true); + assert_eq!(hash_map.get(&42u16), Some(&true)); + assert_eq!(hash_map.get(&43u16), None); + + let mut tree = std::collections::BTreeMap::new(); + tree.insert(ReadOnly::new(42u16), true); + assert_eq!(tree.get(&42u16), Some(&true)); + assert_eq!(tree.get(&43u16), None); + + // Preserve partial orders, including incomparable values. + let nan = ReadOnly::new(f32::NAN); + assert_eq!(nan.partial_cmp(&nan), None); + assert_ne!(nan, nan); + } + + #[test] + fn test_read_only_unsized() { + let bytes = [1u8, 2]; + let value: &ReadOnly<[u8]> = (&bytes[..]).into(); + let as_ref: &[u8] = value.as_ref(); + let borrowed: &[u8] = value.borrow(); + assert_eq!(as_ref, bytes); + assert_eq!(borrowed, bytes); + assert_eq!(value, value); + assert_eq!(value.cmp(value), Ordering::Equal); + + let text: &ReadOnly = "hello".into(); + assert_eq!(format!("{:.3}", text), "hel"); + } + + #[test] + fn test_read_only_formatting() { + let value = ReadOnly::new(42u16); + // Check that formatting flags are forwarded as well as the value. + assert_eq!(format!("{:04?}", value), "0042"); + assert_eq!(format!("{:04}", value), "0042"); + assert_eq!(format!("{:#010b}", value), "0b00101010"); + assert_eq!(format!("{:#06o}", value), "0o0052"); + assert_eq!(format!("{:#06x}", value), "0x002a"); + assert_eq!(format!("{:#06X}", value), "0x002A"); + assert_eq!(format!("{:.2e}", value), "4.20e1"); + assert_eq!(format!("{:.2E}", value), "4.20E1"); + } + #[test] fn test_unalign() { // Test methods that don't depend on alignment. diff --git a/samples/configfs/configfs_sample.c b/samples/configfs/configfs_sample.c index c1b108ec4ea02e..6e3a8e0061aba7 100644 --- a/samples/configfs/configfs_sample.c +++ b/samples/configfs/configfs_sample.c @@ -84,7 +84,7 @@ CONFIGFS_ATTR_RO(childless_, showme); CONFIGFS_ATTR(childless_, storeme); CONFIGFS_ATTR_RO(childless_, description); -static struct configfs_attribute *childless_attrs[] = { +static const struct configfs_attribute *const childless_attrs[] = { &childless_attr_showme, &childless_attr_storeme, &childless_attr_description, @@ -92,7 +92,7 @@ static struct configfs_attribute *childless_attrs[] = { }; static const struct config_item_type childless_type = { - .ct_attrs = childless_attrs, + .ct_attrs_const = childless_attrs, .ct_owner = THIS_MODULE, }; @@ -148,7 +148,7 @@ static ssize_t simple_child_storeme_store(struct config_item *item, CONFIGFS_ATTR(simple_child_, storeme); -static struct configfs_attribute *simple_child_attrs[] = { +static const struct configfs_attribute *const simple_child_attrs[] = { &simple_child_attr_storeme, NULL, }; @@ -164,7 +164,7 @@ static const struct configfs_item_operations simple_child_item_ops = { static const struct config_item_type simple_child_type = { .ct_item_ops = &simple_child_item_ops, - .ct_attrs = simple_child_attrs, + .ct_attrs_const = simple_child_attrs, .ct_owner = THIS_MODULE, }; @@ -205,7 +205,7 @@ static ssize_t simple_children_description_show(struct config_item *item, CONFIGFS_ATTR_RO(simple_children_, description); -static struct configfs_attribute *simple_children_attrs[] = { +static const struct configfs_attribute *const simple_children_attrs[] = { &simple_children_attr_description, NULL, }; @@ -230,7 +230,7 @@ static const struct configfs_group_operations simple_children_group_ops = { static const struct config_item_type simple_children_type = { .ct_item_ops = &simple_children_item_ops, .ct_group_ops = &simple_children_group_ops, - .ct_attrs = simple_children_attrs, + .ct_attrs_const = simple_children_attrs, .ct_owner = THIS_MODULE, }; @@ -283,7 +283,7 @@ static ssize_t group_children_description_show(struct config_item *item, CONFIGFS_ATTR_RO(group_children_, description); -static struct configfs_attribute *group_children_attrs[] = { +static const struct configfs_attribute *const group_children_attrs[] = { &group_children_attr_description, NULL, }; @@ -298,7 +298,7 @@ static const struct configfs_group_operations group_children_group_ops = { static const struct config_item_type group_children_type = { .ct_group_ops = &group_children_group_ops, - .ct_attrs = group_children_attrs, + .ct_attrs_const = group_children_attrs, .ct_owner = THIS_MODULE, }; @@ -313,6 +313,126 @@ static struct configfs_subsystem group_children_subsys = { /* ----------------------------------------------------------------- */ +/* + * 04-symlink-children + * + * This example has children that are valid sources for symlink(2). A + * child accepts a link to any other config_item and reports how many + * links it currently holds, so ->allow_link() and ->drop_link() are + * observable from userspace. + */ + +struct symlink_child { + struct config_item item; + int nlinks; +}; + +static inline struct symlink_child *to_symlink_child(struct config_item *item) +{ + return container_of(item, struct symlink_child, item); +} + +static ssize_t symlink_child_nlinks_show(struct config_item *item, char *page) +{ + return sprintf(page, "%d\n", to_symlink_child(item)->nlinks); +} + +CONFIGFS_ATTR_RO(symlink_child_, nlinks); + +static struct configfs_attribute *symlink_child_attrs[] = { + &symlink_child_attr_nlinks, + NULL, +}; + +/* + * The VFS holds the source item's directory locked across symlink(2) and + * unlink(2), so ->nlinks needs no lock of its own. + */ +static int symlink_child_allow_link(struct config_item *src, + struct config_item *target) +{ + to_symlink_child(src)->nlinks++; + + return 0; +} + +static void symlink_child_drop_link(struct config_item *src, + struct config_item *target) +{ + to_symlink_child(src)->nlinks--; +} + +static void symlink_child_release(struct config_item *item) +{ + kfree(to_symlink_child(item)); +} + +static const struct configfs_item_operations symlink_child_item_ops = { + .release = symlink_child_release, + .allow_link = symlink_child_allow_link, + .drop_link = symlink_child_drop_link, +}; + +static const struct config_item_type symlink_child_type = { + .ct_item_ops = &symlink_child_item_ops, + .ct_attrs = symlink_child_attrs, + .ct_owner = THIS_MODULE, +}; + +static struct config_item *symlink_children_make_item( + struct config_group *group, const char *name) +{ + struct symlink_child *symlink_child; + + symlink_child = kzalloc_obj(*symlink_child, GFP_KERNEL); + if (!symlink_child) + return ERR_PTR(-ENOMEM); + + config_item_init_type_name(&symlink_child->item, name, + &symlink_child_type); + + return &symlink_child->item; +} + +static ssize_t symlink_children_description_show(struct config_item *item, + char *page) +{ + return sprintf(page, +"[04-symlink-children]\n" +"\n" +"This subsystem allows the creation of child config_items that\n" +"symlink(2) can point at other config_items from. Each child\n" +"reports the number of links it holds.\n"); +} + +CONFIGFS_ATTR_RO(symlink_children_, description); + +static struct configfs_attribute *symlink_children_attrs[] = { + &symlink_children_attr_description, + NULL, +}; + +static const struct configfs_group_operations symlink_children_group_ops = { + .make_item = symlink_children_make_item, +}; + +static const struct config_item_type symlink_children_type = { + .ct_group_ops = &symlink_children_group_ops, + .ct_attrs = symlink_children_attrs, + .ct_owner = THIS_MODULE, +}; + +static struct configfs_subsystem symlink_children_subsys = { + .su_group = { + .cg_item = { + .ci_namebuf = "04-symlink-children", + .ci_type = &symlink_children_type, + }, + }, +}; + +/* ----------------------------------------------------------------- */ + /* * We're now done with our subsystem definitions. * For convenience in this module, here's a list of them all. It @@ -324,6 +444,7 @@ static struct configfs_subsystem *example_subsys[] = { &childless_subsys.subsys, &simple_children_subsys, &group_children_subsys, + &symlink_children_subsys, NULL, }; diff --git a/scripts/.gitignore b/scripts/.gitignore index 4215c2208f7e41..70045654391074 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only /asn1_compiler +/gen_init_cpio /gen_packed_field_checks /generate_rust_target /insert-sys-cert diff --git a/scripts/Kconfig.include b/scripts/Kconfig.include index fc10671c297cf3..a7d9201acfcce7 100644 --- a/scripts/Kconfig.include +++ b/scripts/Kconfig.include @@ -23,13 +23,33 @@ success = $(if-success,$(1),y,n) # Return n if exits with 0, y otherwise failure = $(if-success,$(1),n,y) +# $(cc-probe,) +# The command $(cc-option) and $(cc-option-str) test with. Not meant +# to be called directly; use one of those two instead. +cc-probe = trap "rm -rf .tmp_$$" EXIT; mkdir .tmp_$$; $(CC) -Werror $(CLANG_FLAGS) $(shell,echo $(1) | sed "s;^-Wno-;-W;") -c -x c /dev/null -o .tmp_$$/tmp.o + # $(cc-option,) # Return y if the compiler supports , n otherwise -cc-option = $(success,trap "rm -rf .tmp_$$" EXIT; mkdir .tmp_$$; $(CC) -Werror $(CLANG_FLAGS) $(1) -c -x c /dev/null -o .tmp_$$/tmp.o) +cc-option = $(success,$(cc-probe,$(1))) + +# $(cc-option-str,[,]) +# Return if the compiler supports it, otherwise. +# defaults to an empty string and is returned untested. +cc-option-str = $(if-success,$(cc-probe,$(1)),$(1),$(2)) + +# $(ld-probe,) +# The command $(ld-option) and $(ld-option-str) test with. Not meant +# to be called directly; use one of those two instead. +ld-probe = $(LD) -v $(1) # $(ld-option,) # Return y if the linker supports , n otherwise -ld-option = $(success,$(LD) -v $(1)) +ld-option = $(success,$(ld-probe,$(1))) + +# $(ld-option-str,[,]) +# Return if the linker supports it, otherwise. +# defaults to an empty string and is returned untested. +ld-option-str = $(if-success,$(ld-probe,$(1)),$(1),$(2)) # $(as-instr,) # Return y if the assembler supports , n otherwise @@ -61,9 +81,8 @@ ld-version := $(shell,set -- $(ld-info) && echo $2) # machine bit flags # $(m32-flag): -m32 if the compiler supports it, or an empty string otherwise. # $(m64-flag): -m64 if the compiler supports it, or an empty string otherwise. -cc-option-bit = $(if-success,$(CC) -Werror $(1) -E -x c /dev/null -o /dev/null,$(1)) -m32-flag := $(cc-option-bit,-m32) -m64-flag := $(cc-option-bit,-m64) +m32-flag := $(cc-option-str,-m32) +m64-flag := $(cc-option-str,-m64) # Test whether the compiler can link userspace applications cc_can_link_user = $(success,$(srctree)/scripts/cc-can-link.sh $(CC) $(CLANG_FLAGS) $(USERCFLAGS) $(USERLDFLAGS) $(1)) diff --git a/scripts/Kconfig.toolchain b/scripts/Kconfig.toolchain new file mode 100644 index 00000000000000..4b962831d62af1 --- /dev/null +++ b/scripts/Kconfig.toolchain @@ -0,0 +1,274 @@ +# SPDX-License-Identifier: GPL-2.0-only +config CC_VERSION_TEXT + string + default "$(CC_VERSION_TEXT)" + help + This is used in unclear ways: + + - Re-run Kconfig when the compiler is updated + The 'default' property references the environment variable, + CC_VERSION_TEXT so it is recorded in include/config/auto.conf.cmd. + When the compiler is updated, Kconfig will be invoked. + + - Ensure full rebuild when the compiler is updated + include/linux/compiler-version.h contains this option in the comment + line so fixdep adds include/config/CC_VERSION_TEXT into the + auto-generated dependency. When the compiler is updated, syncconfig + will touch it and then every file will be rebuilt. + +config CC_IS_GCC + def_bool $(success,test "$(cc-name)" = GCC) + +config GCC_VERSION + int + default $(cc-version) if CC_IS_GCC + default 0 + +config CC_IS_CLANG + def_bool $(success,test "$(cc-name)" = Clang) + +config CLANG_VERSION + int + default $(cc-version) if CC_IS_CLANG + default 0 + +config AS_IS_GNU + def_bool $(success,test "$(as-name)" = GNU) + +config AS_IS_LLVM + def_bool $(success,test "$(as-name)" = LLVM) + +config AS_VERSION + int + # Use clang version if this is the integrated assembler + default CLANG_VERSION if AS_IS_LLVM + default $(as-version) + +config LD_IS_BFD + def_bool $(success,test "$(ld-name)" = BFD) + +config LD_VERSION + int + default $(ld-version) if LD_IS_BFD + default 0 + +config LD_IS_LLD + def_bool $(success,test "$(ld-name)" = LLD) + +config LLD_VERSION + int + default $(ld-version) if LD_IS_LLD + default 0 + +config RUSTC_VERSION + int + default $(rustc-version) + help + It does not depend on `RUST` since that one may need to use the version + in a `depends on`. + +config RUST_IS_AVAILABLE + def_bool $(success,$(srctree)/scripts/rust_is_available.sh) + help + This shows whether a suitable Rust toolchain is available (found). + + Please see Documentation/rust/quick-start.rst for instructions on how + to satisfy the build requirements of Rust support. + + In particular, the Makefile target 'rustavailable' is useful to check + why the Rust toolchain is not being detected. + +config RUSTC_LLVM_VERSION + int + default $(rustc-llvm-version) + +config RUSTC_LLVM_MAJOR_VERSION + int + default $(shell,expr $(rustc-llvm-version) / 10000) + +config RUSTC_CLANG_LLVM_COMPATIBLE + bool + default y if CC_IS_CLANG && RUSTC_LLVM_MAJOR_VERSION = $(shell,expr $(cc-version) / 10000) + help + This indicates whether Rust and Clang use LLVM of the same major + version. + + Operations involving handling LLVM IR or bitcode (e.g. cross-language + LTO) require the same LLVM major version to work properly. For best + compatibility it is recommended that the exact same LLVM is used. + +config ARCH_HAS_CC_CAN_LINK + bool + +config CC_CAN_LINK + bool + default ARCH_CC_CAN_LINK if ARCH_HAS_CC_CAN_LINK + default $(cc_can_link_user,$(m64-flag)) if 64BIT + default $(cc_can_link_user,$(m32-flag)) + +# Fixed in GCC 14, 13.3, 12.4 and 11.5 +# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 +config GCC_ASM_GOTO_OUTPUT_BROKEN + bool + depends on CC_IS_GCC + default y if GCC_VERSION < 110500 + default y if GCC_VERSION >= 120000 && GCC_VERSION < 120400 + default y if GCC_VERSION >= 130000 && GCC_VERSION < 130300 + +config CC_HAS_ASM_GOTO_OUTPUT + def_bool y + depends on !GCC_ASM_GOTO_OUTPUT_BROKEN + depends on $(success,echo 'int foo(int x) { asm goto ("": "=r"(x) ::: bar); return x; bar: return 0; }' | $(CC) -x c - -c -o /dev/null) + +config CC_HAS_ASM_GOTO_TIED_OUTPUT + depends on CC_HAS_ASM_GOTO_OUTPUT + # Detect buggy gcc and clang, fixed in gcc-11 clang-14. + def_bool $(success,echo 'int foo(int *x) { asm goto (".long (%l[bar]) - .": "+m"(*x) ::: bar); return *x; bar: return 0; }' | $CC -x c - -c -o /dev/null) + +config TOOLS_SUPPORT_RELR + def_bool $(success,env "CC=$(CC)" "LD=$(LD)" "NM=$(NM)" "OBJCOPY=$(OBJCOPY)" $(srctree)/scripts/tools-support-relr.sh) + +config CC_HAS_ASM_INLINE + def_bool $(success,echo 'void foo(void) { asm inline (""); }' | $(CC) -x c - -c -o /dev/null) + +config CC_HAS_ASSUME + bool + # clang needs to be at least 19.1.0 since the meaning of the assume + # attribute changed: + # https://github.com/llvm/llvm-project/commit/c44fa3e8a9a44c2e9a575768a3c185354b9f6c17 + default y if CC_IS_CLANG && CLANG_VERSION >= 190100 + # supported since gcc 13.1.0 + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106654 + default y if CC_IS_GCC && GCC_VERSION >= 130100 + +config CC_HAS_NO_PROFILE_FN_ATTR + def_bool $(success,echo '__attribute__((no_profile_instrument_function)) int x();' | $(CC) -x c - -c -o /dev/null -Werror) + +config CC_OPT_ZERO_INIT_PADDING_BITS + def_string "$(cc-option-str,-fzero-init-padding-bits=all)" + +config CC_OPT_NO_ADDRSIG + def_string "$(cc-option-str,-fno-addrsig)" + +config CC_OPT_DIAGNOSTICS_SHOW_CONTEXT + def_string "$(cc-option-str,-fdiagnostics-show-context=2)" + +config CC_OPT_DIAGNOSTICS_SHOW_INLINING_CHAIN + def_string "$(cc-option-str,-fdiagnostics-show-inlining-chain)" + +config CC_OPT_STRICT_FLEX_ARRAYS + def_string "$(cc-option-str,-fstrict-flex-arrays=3)" + +# gcc-10 renamed --param=allow-store-data-races=0 to -fno-allow-store-data-races +config CC_OPT_ALLOW_STORE_DATA_RACES_PARAM + def_string "$(cc-option-str,--param=allow-store-data-races=0)" + +config CC_OPT_NO_ALLOW_STORE_DATA_RACES + def_string "$(cc-option-str,-fno-allow-store-data-races)" + +config CC_HAS_MNOP_MCOUNT + def_bool $(cc-option,-mnop-mcount) + +# s390-linux-gnu-gcc did not support -mfentry until gcc-9. +config CC_HAS_MFENTRY + def_bool $(cc-option,-mfentry) + +config LD_OPT_FORCE_GROUP_ALLOCATION + def_string "$(ld-option-str,--force-group-allocation)" + +config LD_OPT_NO_WARN_RWX_SEGMENTS + def_string "$(ld-option-str,--no-warn-rwx-segments)" + +config CC_OPT_WNO_ADDRESS_OF_PACKED_MEMBER + def_string "$(cc-option-str,-Wno-address-of-packed-member)" + +config CC_OPT_WNO_FORMAT_OVERFLOW_NON_KPRINTF + def_string "$(cc-option-str,-Wno-format-overflow-non-kprintf)" + +config CC_OPT_WNO_FORMAT_TRUNCATION_NON_KPRINTF + def_string "$(cc-option-str,-Wno-format-truncation-non-kprintf)" + +config CC_OPT_WNO_DEFAULT_CONST_INIT_UNSAFE + def_string "$(cc-option-str,-Wno-default-const-init-unsafe)" + +config CC_OPT_WNO_DANGLING_POINTER + def_string "$(cc-option-str,-Wno-dangling-pointer)" + +config CC_OPT_WVLA_LARGER_THAN + def_string "$(cc-option-str,-Wvla-larger-than=1)" + +config CC_HAS_WSTRINGOP_OVERFLOW + def_bool $(cc-option,-Wstringop-overflow) + +config CC_OPT_WNO_UNTERMINATED_STRING_INITIALIZATION + def_string "$(cc-option-str,-Wno-unterminated-string-initialization)" + +config CC_OPT_WERROR_DESIGNATED_INIT + def_string "$(cc-option-str,-Werror=designated-init)" + +config CC_OPT_WENUM_CONVERSION + def_string "$(cc-option-str,-Wenum-conversion)" + +config CC_OPT_WNO_PACKED_NOT_ALIGNED + def_string "$(cc-option-str,-Wno-packed-not-aligned)" + +config CC_OPT_WNO_FORMAT_OVERFLOW + def_string "$(cc-option-str,-Wno-format-overflow)" + +config CC_OPT_WNO_STRINGOP_TRUNCATION + def_string "$(cc-option-str,-Wno-stringop-truncation)" + +config CC_HAS_COUNTED_BY + bool + # clang needs to be at least 20.1.0 to avoid potential crashes + # when building structures that contain __counted_by + # https://github.com/ClangBuiltLinux/linux/issues/2114 + # https://github.com/llvm/llvm-project/commit/160fb1121cdf703c3ef5e61fb26c5659eb581489 + default y if CC_IS_CLANG && CLANG_VERSION >= 200100 + # supported since gcc 15.1.0 + # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108896 + default y if CC_IS_GCC && GCC_VERSION >= 150100 + +config CC_HAS_COUNTED_BY_PTR + bool + # supported since clang 22 + default y if CC_IS_CLANG && CLANG_VERSION >= 220100 + # supported since gcc 16.0.0 + default y if CC_IS_GCC && GCC_VERSION >= 160000 + +config CC_HAS_BROKEN_COUNTED_BY_REF + bool + # https://github.com/llvm/llvm-project/issues/182575 + default y if CC_IS_CLANG && CLANG_VERSION < 220100 + +config CC_HAS_ALLOC_TOKEN + def_bool $(cc-option,-falloc-token-max=123) + +config CC_HAS_MULTIDIMENSIONAL_NONSTRING + def_bool $(success,echo 'char tag[][4] __attribute__((__nonstring__)) = { };' | $(CC) $(CLANG_FLAGS) -x c - -c -o /dev/null -Werror) + +config CC_OPT_INLINE_MEMSET + string + default "-finline-stringops=memset" if $(cc-option,-finline-stringops=memset) + default "-mllvm -max-store-memset=4294967294" if $(cc-option,-mllvm -max-store-memset=4294967294) + +config LD_CAN_USE_KEEP_IN_OVERLAY + # ld.lld prior to 21.0.0 did not support KEEP within an overlay description + # https://github.com/llvm/llvm-project/pull/130661 + def_bool LD_IS_BFD || LLD_VERSION >= 210000 + +config RUSTC_HAS_UNNECESSARY_TRANSMUTES + def_bool RUSTC_VERSION >= 108800 + +config RUSTC_HAS_FILE_WITH_NUL + def_bool RUSTC_VERSION >= 108900 + +config RUSTC_HAS_FILE_AS_C_STR + def_bool RUSTC_VERSION >= 109100 + +config RUSTC_HAS_SUSPICIOUS_RUNTIME_SYMBOL_DEFINITIONS + def_bool RUSTC_VERSION >= 109800 + +config PAHOLE_VERSION + int + default "$(PAHOLE_VERSION)" diff --git a/scripts/Makefile b/scripts/Makefile index 3434a82a119f09..d7fbb7ca098b18 100644 --- a/scripts/Makefile +++ b/scripts/Makefile @@ -3,7 +3,7 @@ # scripts contains sources for various helper programs used throughout # the kernel for the build process. -hostprogs-always-$(CONFIG_KALLSYMS) += kallsyms +hostprogs-always-y += kallsyms hostprogs-always-$(BUILD_C_RECORDMCOUNT) += recordmcount hostprogs-always-$(CONFIG_BUILDTIME_TABLE_SORT) += sorttable hostprogs-always-$(CONFIG_ASN1) += asn1_compiler @@ -13,6 +13,7 @@ hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS) += rustdoc_test_builder hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS) += rustdoc_test_gen hostprogs-always-$(CONFIG_TRACEPOINTS) += tracepoint-update +kallsyms-objs := kallsyms.o kallsyms-sysmap.o elf-parse.o sorttable-objs := sorttable.o elf-parse.o tracepoint-update-objs := tracepoint-update.o elf-parse.o @@ -30,6 +31,8 @@ rustdoc_test_builder-rust := y rustdoc_test_gen-rust := y HOSTCFLAGS_tracepoint-update.o = -I$(srctree)/tools/include +HOSTCFLAGS_kallsyms.o = -I$(srctree)/tools/include +HOSTCFLAGS_kallsyms-sysmap.o = -I$(srctree)/tools/include HOSTCFLAGS_elf-parse.o = -I$(srctree)/tools/include HOSTCFLAGS_sorttable.o = -I$(srctree)/tools/include HOSTLDLIBS_sorttable = -lpthread @@ -53,6 +56,9 @@ ifdef CONFIG_BUILDTIME_MCOUNT_SORT HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED endif +hostprogs += gen_init_cpio +always-$(CONFIG_BLK_DEV_INITRD) += gen_init_cpio + # The following programs are only built on demand hostprogs += unifdef gen_packed_field_checks diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo index 1442043da1391b..57a3aba6ffc237 100644 --- a/scripts/Makefile.autofdo +++ b/scripts/Makefile.autofdo @@ -22,7 +22,7 @@ ifdef CONFIG_LTO_CLANG_THIN KBUILD_LDFLAGS += --lto-sample-profile=$(CLANG_AUTOFDO_PROFILE) endif KBUILD_LDFLAGS += --mllvm=-enable-fs-discriminator=true --mllvm=-improved-fs-discriminator=true -plugin-opt=thinlto - KBUILD_LDFLAGS += -plugin-opt=-split-machine-functions + KBUILD_LDFLAGS += $(call ld-option,-plugin-opt=-function-splitting=all,-plugin-opt=-split-machine-functions) endif export CFLAGS_AUTOFDO_CLANG RUSTFLAGS_AUTOFDO_CLANG diff --git a/scripts/Makefile.build b/scripts/Makefile.build index 4349108e75e1f7..6127feb970c650 100644 --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -306,6 +306,27 @@ quiet_cmd_cc_lst_c = MKLST $@ $(obj)/%.lst: $(obj)/%.c FORCE $(call if_changed_dep,cc_lst_c) +# Compile C headers (.h) for header check +# --------------------------------------------------------------------------- + +# Check a header is self-contained, has header guards, and passes kernel-doc. +# +# This is for 'make HEADER_CHECK=(headers|dirs) headercheck'. Not to be confused +# with CONFIG_UAPI_HEADER_TEST. +# +# Please do *not* plug %.header-check into any configurable targets, unless you +# also figure out a way to completely hide the %.header-check artefacts. + +quiet_cmd_header_check = HDRCHK $(patsubst %.header-check,%.h,$@) + cmd_header_check = \ + $(CC) $(c_flags) -fsyntax-only -x c /dev/null -include $< -include $<; \ + PYTHONDONTWRITEBYTECODE=1 $(PYTHON3) $(KERNELDOC) -none $(KDOCFLAGS) \ + $(if $(findstring e, $(KBUILD_EXTRA_WARN))$(CONFIG_WERROR),-Werror) $<; \ + touch $@ + +$(obj)/%.header-check: $(obj)/%.h FORCE + $(call if_changed_dep,header_check) + # Compile Rust sources (.rs) # --------------------------------------------------------------------------- diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins index b0e1423b09c213..9b98995805712f 100644 --- a/scripts/Makefile.gcc-plugins +++ b/scripts/Makefile.gcc-plugins @@ -8,6 +8,10 @@ ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY endif export DISABLE_LATENT_ENTROPY_PLUGIN +# Disable recording GCC options in debug info when building external modules to +# avoid leaking absolute host build paths +gcc-plugin-cflags-$(if $(KBUILD_EXTMOD),y) += -gno-record-gcc-switches + # All the plugin CFLAGS are collected here in case a build target needs to # filter them out of the KBUILD_CFLAGS. GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y)) -DGCC_PLUGINS diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 0a4fdd8bd975de..19e96b30a122f7 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -333,11 +333,11 @@ quiet_cmd_ar = AR $@ quiet_cmd_objcopy = OBJCOPY $@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@ -# Gzip +# Gzip, for the kernel image # --------------------------------------------------------------------------- quiet_cmd_gzip = GZIP $@ - cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@ + cmd_gzip = cat $(real-prereqs) | $(KPGZIP) -n -f -9 > $@ # Bzip2 # --------------------------------------------------------------------------- diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 6d36786ba31cef..948ad5b679d110 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -198,7 +198,7 @@ tar%-pkg: linux-$(KERNELRELEASE)-$(ARCH).tar.% FORCE $(Q)$(MAKE) -f $(srctree)/Makefile INSTALL_MOD_PATH=$@/$(INSTALL_MOD_PATH) modules_install quiet_cmd_cpio = CPIO $@ - cmd_cpio = $(CONFIG_SHELL) $(srctree)/usr/gen_initramfs.sh -o $@ $< + cmd_cpio = $(CONFIG_SHELL) $(srctree)/scripts/gen_initramfs.sh -o $@ $< modules-$(KERNELRELEASE)-$(ARCH).cpio: .tmp_modules_cpio $(call cmd,cpio) diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux index fcae1e432d9add..95b523fe4f9eeb 100644 --- a/scripts/Makefile.vmlinux +++ b/scripts/Makefile.vmlinux @@ -89,11 +89,8 @@ remove-section-$(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS) += '.rel.*' remove-symbols := -w --strip-unneeded-symbol='__mod_device_table__*' -# To avoid warnings: "empty loadable segment detected at ..." from GNU objcopy, -# it is necessary to remove the PT_LOAD flag from the segment. quiet_cmd_strip_relocs = OBJCOPY $@ - cmd_strip_relocs = $(OBJCOPY) $(patsubst %,--set-section-flags %=noload,$(remove-section-y)) $< $@; \ - $(OBJCOPY) $(addprefix --remove-section=,$(remove-section-y)) $(remove-symbols) $@ + cmd_strip_relocs = $(OBJCOPY) $(addprefix --remove-section=,$(remove-section-y)) $(remove-symbols) $< $@ targets += vmlinux vmlinux: vmlinux.unstripped FORCE @@ -102,21 +99,8 @@ vmlinux: vmlinux.unstripped FORCE # modules.builtin.modinfo # --------------------------------------------------------------------------- -# .modinfo in vmlinux.unstripped is aligned to 8 bytes for compatibility with -# tools that expect vmlinux to have sufficiently aligned sections but the -# additional bytes used for padding .modinfo to satisfy this requirement break -# certain versions of kmod with -# -# depmod: ERROR: kmod_builtin_iter_next: unexpected string without modname prefix -# -# Strip the trailing padding bytes after extracting .modinfo to comply with -# what kmod expects to parse. quiet_cmd_modules_builtin_modinfo = GEN $@ - cmd_modules_builtin_modinfo = $(cmd_objcopy); \ - sed -i 's/\x00\+$$/\x00/g' $@; \ - chmod -x $@ - -OBJCOPYFLAGS_modules.builtin.modinfo := -j .modinfo -O binary + cmd_modules_builtin_modinfo = $(OBJCOPY) -O binary -j .modinfo --dump-section .modinfo=$@ $< /dev/null targets += modules.builtin.modinfo modules.builtin.modinfo: vmlinux.unstripped FORCE diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o index 24a3a4fd271c23..0b37bd5dc3a491 100644 --- a/scripts/Makefile.vmlinux_o +++ b/scripts/Makefile.vmlinux_o @@ -19,7 +19,7 @@ quiet_cmd_gen_initcalls_lds = GEN $@ .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \ vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE - $(call if_changed,gen_initcalls_lds) + +$(call if_changed,gen_initcalls_lds) targets := .tmp_initcalls.lds diff --git a/scripts/Makefile.warn b/scripts/Makefile.warn index 35af7d6c6d1832..556fa249ca071a 100644 --- a/scripts/Makefile.warn +++ b/scripts/Makefile.warn @@ -17,7 +17,7 @@ KBUILD_CFLAGS += -Werror=strict-prototypes KBUILD_CFLAGS += -Wno-format-security KBUILD_CFLAGS += -Wno-trigraphs KBUILD_CFLAGS += -Wno-frame-address -KBUILD_CFLAGS += $(call cc-option, -Wno-address-of-packed-member) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_ADDRESS_OF_PACKED_MEMBER) KBUILD_CFLAGS += -Wmissing-declarations KBUILD_CFLAGS += -Wmissing-prototypes @@ -30,8 +30,8 @@ KBUILD_CFLAGS-$(CONFIG_CC_NO_ARRAY_BOUNDS) += -Wno-array-bounds ifdef CONFIG_CC_IS_CLANG # Clang checks for overflow/truncation with '%p', while GCC does not: # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111219 -KBUILD_CFLAGS += $(call cc-option, -Wno-format-overflow-non-kprintf) -KBUILD_CFLAGS += $(call cc-option, -Wno-format-truncation-non-kprintf) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_FORMAT_OVERFLOW_NON_KPRINTF) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_FORMAT_TRUNCATION_NON_KPRINTF) # Clang may emit a warning when a const variable, such as the dummy variables # in typecheck(), or const member of an aggregate type are not initialized, @@ -43,7 +43,7 @@ KBUILD_CFLAGS += $(call cc-option, -Wno-format-truncation-non-kprintf) # disabled with this same switch, there should not be too much coverage lost # because -Wuninitialized will still flag when an uninitialized const variable # is used. -KBUILD_CFLAGS += $(call cc-option, -Wno-default-const-init-unsafe) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_DEFAULT_CONST_INIT_UNSAFE) else # gcc inanely warns about local variables called 'main' @@ -54,7 +54,7 @@ endif KBUILD_CFLAGS += -Wno-type-limits # These result in bogus false positives -KBUILD_CFLAGS += $(call cc-option, -Wno-dangling-pointer) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_DANGLING_POINTER) # Stack Variable Length Arrays (VLAs) must not be used in the kernel. # Function array parameters should, however, be usable, but -Wvla will @@ -62,7 +62,7 @@ KBUILD_CFLAGS += $(call cc-option, -Wno-dangling-pointer) # types, so depend on GCC for now to keep stack VLAs out of the tree. # https://github.com/llvm/llvm-project/issues/57098 # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98217 -KBUILD_CFLAGS += $(call cc-option,-Wvla-larger-than=1) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WVLA_LARGER_THAN) # disable pointer signed / unsigned warnings in gcc 4.0 KBUILD_CFLAGS += -Wno-pointer-sign @@ -73,11 +73,11 @@ KBUILD_CFLAGS += -Wno-pointer-sign KBUILD_CFLAGS += -Wcast-function-type # Currently, disable -Wstringop-overflow for GCC 11, globally. -KBUILD_CFLAGS-$(CONFIG_CC_NO_STRINGOP_OVERFLOW) += $(call cc-option, -Wno-stringop-overflow) -KBUILD_CFLAGS-$(CONFIG_CC_STRINGOP_OVERFLOW) += $(call cc-option, -Wstringop-overflow) +KBUILD_CFLAGS-$(CONFIG_CC_NO_STRINGOP_OVERFLOW) += $(if $(CONFIG_CC_HAS_WSTRINGOP_OVERFLOW),-Wno-stringop-overflow) +KBUILD_CFLAGS-$(CONFIG_CC_STRINGOP_OVERFLOW) += $(if $(CONFIG_CC_HAS_WSTRINGOP_OVERFLOW),-Wstringop-overflow) # Currently, disable -Wunterminated-string-initialization as broken -KBUILD_CFLAGS += $(call cc-option, -Wno-unterminated-string-initialization) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_UNTERMINATED_STRING_INITIALIZATION) # The allocators already balk at large sizes, so silence the compiler # warnings for bounds checks involving those possible values. While @@ -100,10 +100,10 @@ KBUILD_CFLAGS += -Werror=date-time KBUILD_CFLAGS += -Werror=incompatible-pointer-types # Require designated initializers for all marked structures -KBUILD_CFLAGS += $(call cc-option,-Werror=designated-init) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WERROR_DESIGNATED_INIT) # Warn if there is an enum types mismatch -KBUILD_CFLAGS += $(call cc-option,-Wenum-conversion) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WENUM_CONVERSION) KBUILD_CFLAGS += -Wunused @@ -125,12 +125,12 @@ else # Suppress them by using -Wno... except for W=1. KBUILD_CFLAGS += -Wno-unused-but-set-variable KBUILD_CFLAGS += -Wno-unused-const-variable -KBUILD_CFLAGS += $(call cc-option, -Wno-packed-not-aligned) -KBUILD_CFLAGS += $(call cc-option, -Wno-format-overflow) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_PACKED_NOT_ALIGNED) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_FORMAT_OVERFLOW) ifdef CONFIG_CC_IS_GCC KBUILD_CFLAGS += -Wno-format-truncation endif -KBUILD_CFLAGS += $(call cc-option, -Wno-stringop-truncation) +KBUILD_CFLAGS += $(CONFIG_CC_OPT_WNO_STRINGOP_TRUNCATION) KBUILD_CFLAGS += -Wno-override-init # alias for -Wno-initializer-overrides in clang diff --git a/scripts/check-function-names.sh b/scripts/check-function-names.sh index 08071133e5a512..94883e690627da 100755 --- a/scripts/check-function-names.sh +++ b/scripts/check-function-names.sh @@ -13,7 +13,8 @@ if [ ! -f "$objfile" ]; then exit 1 fi -bad_symbols=$(${NM:-nm} "$objfile" | awk '$2 ~ /^[TtWw]$/ {print $3}' | grep -E '^(startup|exit|split|unlikely|hot|unknown)(\.|$)') +bad_symbols=$(${NM:-nm} -p "$objfile" | awk '$2 ~ /^[TtWw]$/ {print $3}' | + grep -E '^(startup|exit|split|unlikely|hot|unknown)(\.|$)') if [ -n "$bad_symbols" ]; then echo "$bad_symbols" | while read -r sym; do diff --git a/scripts/checkkconfigsymbols.py b/scripts/checkkconfigsymbols.py index 36c920e7131373..9f7aed98d91c0c 100755 --- a/scripts/checkkconfigsymbols.py +++ b/scripts/checkkconfigsymbols.py @@ -245,7 +245,7 @@ def tree_is_dirty(): """Return true if the current working tree is dirty (i.e., if any file has been added, deleted, modified, renamed or copied but not committed).""" stdout = execute(["git", "status", "--porcelain"]) - for line in stdout: + for line in stdout.splitlines(): if re.findall(r"[URMADC]{1}", line[:2]): return True return False diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index 8a7787d228a63d..c9840f5219c2a9 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -655,7 +655,6 @@ sub hash_show_words { Reviewed-by:| Reported-by:| Suggested-by:| - Assisted-by:| To:| Cc: )}; @@ -3124,16 +3123,6 @@ sub process { } } - # Assisted-by uses a free-form value (e.g. "LLM"), not an - # email address, so skip the email format checks below. - if ($sign_off =~ /^Assisted-by:/i) { - if ($email =~ /^\s*$/) { - WARN("BAD_SIGN_OFF", - "Assisted-by requires a value\n" . $herecurr); - } - next; - } - my ($email_name, $name_comment, $email_address, $comment) = parse_email($email); my $suggested_email = format_email(($email_name, $name_comment, $email_address, $comment)); if ($suggested_email eq "") { diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py index e78be82aa6931b..79d58b34e7acd9 100755 --- a/scripts/clang-tools/run-clang-tools.py +++ b/scripts/clang-tools/run-clang-tools.py @@ -12,6 +12,7 @@ import argparse import json import multiprocessing +import os import subprocess import sys diff --git a/scripts/container b/scripts/container index b05333d8530be5..a3a55180d6ed79 100755 --- a/scripts/container +++ b/scripts/container @@ -1,17 +1,19 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0-only -# Copyright (C) 2025 Guillaume Tucker +# Copyright (C) 2025-2026 Guillaume Tucker """Containerized builds""" import abc import argparse +import dataclasses import logging import os import pathlib import shutil import subprocess import sys +import tomllib import uuid @@ -20,10 +22,17 @@ class ContainerRuntime(abc.ABC): name = None # Property defined in each implementation class - def __init__(self, args, logger): - self._uid = args.uid or os.getuid() - self._gid = args.gid or args.uid or os.getgid() - self._env_file = args.env_file + def __init__(self, args, config, logger): + def _first_not_none(*args): + return next((item for item in args if item is not None)) + + self._uid = _first_not_none( + args.uid, config.uid, os.getuid() + ) + self._gid = _first_not_none( + args.gid, config.gid, args.uid, config.uid, os.getgid() + ) + self._env_file = args.env_file or config.env_file self._shell = args.shell self._logger = logger @@ -131,6 +140,42 @@ class Runtimes: raise ValueError("no runtime found") +@dataclasses.dataclass +class Config: + """Container configuration""" + image: str = None + runtime: str = None + env_file: str = None + uid: int = None + gid: int = None + + @classmethod + def from_toml(cls, config_file_path, profile_name): + """Create a config object from a TOML file""" + if not config_file_path: + config_file_path = '.container.toml' + if not os.path.exists(config_file_path): + return cls() + elif not os.path.exists(config_file_path): + raise ValueError(f"config file not found: {config_file_path}") + with open(config_file_path, 'rb') as config_file: + config = tomllib.load(config_file) + default = config.get('DEFAULT', {}) + if not profile_name: + profile = {} + else: + profile = config.get(profile_name) + if profile is None: + raise ValueError(f"unknown profile: {profile_name}") + kwargs = { + name: type(value) for (name, type, value) in ( + (op.name, op.type, profile.get(op.name, default.get(op.name))) + for op in dataclasses.fields(cls) + ) if value is not None + } + return cls(**kwargs) + + def _get_logger(verbose): """Set up a logger with the appropriate level""" logger = logging.getLogger('container') @@ -147,13 +192,19 @@ def main(args): """Main entry point for the container tool""" logger = _get_logger(args.verbose) try: - cls = Runtimes.get(args.runtime) if args.runtime else Runtimes.find() + config = Config.from_toml(args.config_file, args.config_profile) + runtime = args.runtime or config.runtime + cls = Runtimes.get(runtime) if runtime else Runtimes.find() except ValueError as ex: logger.error(ex) return 1 logger.debug("runtime: %s", cls.name) - logger.debug("image: %s", args.image) - return cls(args, logger).run(args.image, args.cmd) + image = args.image or config.image + if not image: + logger.error("no image specified") + return 1 + logger.debug("image: %s", image) + return cls(args, config, logger).run(image, args.cmd) if __name__ == '__main__': @@ -162,18 +213,28 @@ if __name__ == '__main__': description="See the documentation for more details: " "https://docs.kernel.org/dev-tools/container.html" ) + parser.add_argument( + '-c', '--config-file', + help="Path to the config file. If not specified, the default is to " + "look for .container.toml in the current working directory." + ) parser.add_argument( '-e', '--env-file', help="Path to an environment file to load in the container." ) parser.add_argument( - '-g', '--gid', + '-g', '--gid', type=int, help="Group ID to use inside the container." ) parser.add_argument( - '-i', '--image', required=True, + '-i', '--image', help="Container image name." ) + parser.add_argument( + '-p', '--config-profile', + help="Profile section to use in the config file. This will override " + "any values defined in the DEFAULT section." + ) parser.add_argument( '-r', '--runtime', choices=Runtimes.get_names(), help="Container runtime name. If not specified, the first one found " @@ -184,9 +245,10 @@ if __name__ == '__main__': help="Run the container in an interactive shell." ) parser.add_argument( - '-u', '--uid', + '-u', '--uid', type=int, help="User ID to use inside the container. If the -g option is not " - "specified, the user ID will also be set as the group ID." + "specified and no group ID is defined in the configuration file, the " + "user ID will also be set as the group ID." ) parser.add_argument( '-v', '--verbose', action='store_true', diff --git a/scripts/elf-parse.c b/scripts/elf-parse.c index 99869ff91a8c7f..ce1c2a670244eb 100644 --- a/scripts/elf-parse.c +++ b/scripts/elf-parse.c @@ -17,15 +17,21 @@ struct elf_funcs elf_parser; * Get the whole file as a programming convenience in order to avoid * malloc+lseek+read+free of many pieces. If successful, then mmap * avoids copying unused pieces; else just read the whole file. - * Open for both read and write. + * Open for both read and write if writable is true, otherwise open + * read-only. */ -static void *map_file(char const *fname, size_t *size) +static void *map_file(char const *fname, size_t *size, bool writable) { - int fd; + int fd, prot = PROT_READ, flags = MAP_PRIVATE; struct stat sb; void *addr = NULL; - fd = open(fname, O_RDWR); + if (writable) { + prot |= PROT_WRITE; + flags = MAP_SHARED; + } + + fd = open(fname, writable ? O_RDWR : O_RDONLY); if (fd < 0) { perror(fname); return NULL; @@ -39,7 +45,7 @@ static void *map_file(char const *fname, size_t *size) goto out; } - addr = mmap(0, sb.st_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); + addr = mmap(0, sb.st_size, prot, flags, fd, 0); if (addr == MAP_FAILED) { fprintf(stderr, "Could not mmap file: %s\n", fname); goto out; @@ -57,6 +63,12 @@ static int elf_parse(const char *fname, void *addr, uint32_t types) Elf_Ehdr *ehdr = addr; uint16_t type; + if (memcmp(ELFMAG, ehdr->e32.e_ident, SELFMAG) != 0 || + ehdr->e32.e_ident[EI_VERSION] != EV_CURRENT) { + fprintf(stderr, "unrecognized ELF file %s\n", fname); + return -1; + } + switch (ehdr->e32.e_ident[EI_DATA]) { case ELFDATA2LSB: elf_parser.r = rle; @@ -78,12 +90,6 @@ static int elf_parse(const char *fname, void *addr, uint32_t types) return -1; } - if (memcmp(ELFMAG, ehdr->e32.e_ident, SELFMAG) != 0 || - ehdr->e32.e_ident[EI_VERSION] != EV_CURRENT) { - fprintf(stderr, "unrecognized ELF file %s\n", fname); - return -1; - } - type = elf_parser.r2(&ehdr->e32.e_type); if (!((1 << type) & types)) { fprintf(stderr, "Invalid ELF type file %s\n", fname); @@ -103,7 +109,9 @@ static int elf_parse(const char *fname, void *addr, uint32_t types) elf_parser.shdr_name = shdr32_name; elf_parser.shdr_type = shdr32_type; elf_parser.shdr_entsize = shdr32_entsize; + elf_parser.shdr_flags = shdr32_flags; elf_parser.sym_type = sym32_type; + elf_parser.sym_bind = sym32_bind; elf_parser.sym_name = sym32_name; elf_parser.sym_value = sym32_value; elf_parser.sym_shndx = sym32_shndx; @@ -133,7 +141,9 @@ static int elf_parse(const char *fname, void *addr, uint32_t types) elf_parser.shdr_name = shdr64_name; elf_parser.shdr_type = shdr64_type; elf_parser.shdr_entsize = shdr64_entsize; + elf_parser.shdr_flags = shdr64_flags; elf_parser.sym_type = sym64_type; + elf_parser.sym_bind = sym64_bind; elf_parser.sym_name = sym64_name; elf_parser.sym_value = sym64_value; elf_parser.sym_shndx = sym64_shndx; @@ -174,12 +184,13 @@ int elf_map_long_size(void *addr) return ehdr->e32.e_ident[EI_CLASS] == ELFCLASS32 ? 4 : 8; } -void *elf_map(char const *fname, size_t *size, uint32_t types) +static void *__elf_map(char const *fname, size_t *size, uint32_t types, + bool writable) { void *addr; int ret; - addr = map_file(fname, size); + addr = map_file(fname, size, writable); if (!addr) return NULL; @@ -192,7 +203,73 @@ void *elf_map(char const *fname, size_t *size, uint32_t types) return addr; } +void *elf_map(char const *fname, size_t *size, uint32_t types) +{ + return __elf_map(fname, size, types, true); +} + +/* For tools that only read the file. */ +void *elf_map_ro(char const *fname, size_t *size, uint32_t types) +{ + return __elf_map(fname, size, types, false); +} + void elf_unmap(void *addr, size_t size) { munmap(addr, size); } + +/* + * Open an ELF file and map it read-only, locating its section headers, section + * names and symbol table which are populated in the elf out parameter. + * + * Returns 0 on success, otherwise -1. + */ +int elf_open_ro(char const *fname, uint32_t types, struct elf_file *elf) +{ + Elf_Ehdr *ehdr; + Elf_Shdr *first; + unsigned int shstrndx, i; + + memset(elf, 0, sizeof(*elf)); + elf->base = elf_map_ro(fname, &elf->size, types); + if (!elf->base) + return -1; + + ehdr = elf->base; + elf->shdrs = (const char *)elf->base + ehdr_shoff(ehdr); + elf->shentsize = ehdr_shentsize(ehdr); + first = elf_section(elf, 0); + + /* A count or index too large for the header lives in section 0. */ + elf->shnum = ehdr_shnum(ehdr); + if (elf->shnum == SHN_UNDEF) + elf->shnum = shdr_size(first); + shstrndx = ehdr_shstrndx(ehdr); + if (shstrndx == SHN_XINDEX) + shstrndx = shdr_link(first); + elf->shstrtab = (const char *)elf->base + + shdr_offset(elf_section(elf, shstrndx)); + + for (i = 0; i < elf->shnum && !elf->symtab; i++) + if (shdr_type(elf_section(elf, i)) == SHT_SYMTAB) + elf->symtab = elf_section(elf, i); + + if (!elf->symtab) { + fprintf(stderr, "%s: no symbol table\n", fname); + elf_close(elf); + return -1; + } + + elf->strtab = (const char *)elf->base + + shdr_offset(elf_section(elf, shdr_link(elf->symtab))); + elf->nr_syms = shdr_size(elf->symtab) / shdr_entsize(elf->symtab); + + return 0; +} + +void elf_close(struct elf_file *elf) +{ + elf_unmap(elf->base, elf->size); + elf->base = NULL; +} diff --git a/scripts/elf-parse.h b/scripts/elf-parse.h index f4411e03069dc5..c5ad754da25403 100644 --- a/scripts/elf-parse.h +++ b/scripts/elf-parse.h @@ -37,10 +37,12 @@ struct elf_funcs { uint64_t (*shdr_offset)(Elf_Shdr *shdr); uint64_t (*shdr_size)(Elf_Shdr *shdr); uint64_t (*shdr_entsize)(Elf_Shdr *shdr); + uint64_t (*shdr_flags)(Elf_Shdr *shdr); uint32_t (*shdr_link)(Elf_Shdr *shdr); uint32_t (*shdr_name)(Elf_Shdr *shdr); uint32_t (*shdr_type)(Elf_Shdr *shdr); uint8_t (*sym_type)(Elf_Sym *sym); + uint8_t (*sym_bind)(Elf_Sym *sym); uint32_t (*sym_name)(Elf_Sym *sym); uint64_t (*sym_value)(Elf_Sym *sym); uint16_t (*sym_shndx)(Elf_Sym *sym); @@ -143,6 +145,7 @@ SHDR_ADDR(addr) SHDR_ADDR(offset) SHDR_ADDR(size) SHDR_ADDR(entsize) +SHDR_ADDR(flags) SHDR_WORD(link) SHDR_WORD(name) @@ -211,6 +214,21 @@ static inline uint8_t sym_type(Elf_Sym *sym) return elf_parser.sym_type(sym); } +static inline uint8_t sym64_bind(Elf_Sym *sym) +{ + return ELF64_ST_BIND(sym->e64.st_info); +} + +static inline uint8_t sym32_bind(Elf_Sym *sym) +{ + return ELF32_ST_BIND(sym->e32.st_info); +} + +static inline uint8_t sym_bind(Elf_Sym *sym) +{ + return elf_parser.sym_bind(sym); +} + SYM_ADDR(value) SYM_WORD(name) SYM_HALF(shndx) @@ -298,8 +316,44 @@ static inline void w8le(uint64_t val, uint64_t *x) } void *elf_map(char const *fname, size_t *size, uint32_t types); +void *elf_map_ro(char const *fname, size_t *size, uint32_t types); void elf_unmap(void *addr, size_t size); int elf_map_machine(void *addr); int elf_map_long_size(void *addr); +/* A mapped file with its section headers, section names and symbol table. */ +struct elf_file { + void *base; + size_t size; + const char *shdrs; + unsigned int shnum, shentsize; + const char *shstrtab; + Elf_Shdr *symtab; + const char *strtab; + size_t nr_syms; +}; + +int elf_open_ro(char const *fname, uint32_t types, struct elf_file *elf); +void elf_close(struct elf_file *elf); + +static inline Elf_Shdr *elf_section(const struct elf_file *elf, + unsigned int index) +{ + return (Elf_Shdr *)(elf->shdrs + (size_t)index * elf->shentsize); +} + +static inline const char *elf_section_name(const struct elf_file *elf, + Elf_Shdr *shdr) +{ + return elf->shstrtab + shdr_name(shdr); +} + +static inline Elf_Sym *elf_symbol(const struct elf_file *elf, size_t index) +{ + const char *base = elf->base; + + return (Elf_Sym *)(base + shdr_offset(elf->symtab) + + index * shdr_entsize(elf->symtab)); +} + #endif /* _SCRIPTS_ELF_PARSE_H */ diff --git a/usr/gen_init_cpio.c b/scripts/gen_init_cpio.c similarity index 99% rename from usr/gen_init_cpio.c rename to scripts/gen_init_cpio.c index b7296edc6626fb..0f0c61ec21ca3d 100644 --- a/usr/gen_init_cpio.c +++ b/scripts/gen_init_cpio.c @@ -756,6 +756,7 @@ int main (int argc, char *argv[]) "ERROR: incorrect format, newline required line %d: '%s'\n", line_nr, line); ec = -1; + break; } for (type_idx = 0; file_handler_table[type_idx].type; type_idx++) { diff --git a/usr/gen_initramfs.sh b/scripts/gen_initramfs.sh similarity index 97% rename from usr/gen_initramfs.sh rename to scripts/gen_initramfs.sh index 7eba2fddf0ef2e..9c6168a9338d44 100755 --- a/usr/gen_initramfs.sh +++ b/scripts/gen_initramfs.sh @@ -6,7 +6,7 @@ # # Generate a cpio packed initramfs. It uses gen_init_cpio to generate # the cpio archive. -# This script assumes that gen_init_cpio is located in usr/ directory +# This script assumes that gen_init_cpio is located in scripts/ directory # error out on errors set -e @@ -247,4 +247,4 @@ done # If output_file is set we will generate cpio archive # we are careful to delete tmp files -usr/gen_init_cpio $output $timestamp $cpio_list +scripts/gen_init_cpio $output $timestamp $cpio_list diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 21b319e6c9a5f5..89119acb1b9e22 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -4,7 +4,8 @@ """ Determines how many parallel tasks "make" is expecting, as it is not exposed via any special variables, reserves them all, runs a subprocess -with PARALLELISM environment variable set, and releases the jobs back again. +with PARALLELISM environment variable set and any %PARALLELISM% in its +arguments replaced by the count, and releases the jobs back again. See: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver diff --git a/scripts/kallsyms-sysmap.c b/scripts/kallsyms-sysmap.c new file mode 100644 index 00000000000000..a482968eba5112 --- /dev/null +++ b/scripts/kallsyms-sysmap.c @@ -0,0 +1,248 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Obtain symbols from vmlinux for usage by kallsyms. Replaces mksysmap. + * + * To retain compatibility, it provides the same output as nm, only faster. + */ + +#include +#include +#include +#include +#include + +#include "elf-parse.h" +#include "kallsyms.h" + +/* + * What mksysmap dropped from System.map, by name: symbols that are not needed + * there, or not suitable for kallsyms. + */ +static const char *const sysmap_omit_prefixes[] = { + /* local symbols for ARM, MIPS, etc. */ + "$", + /* local labels, .LBB, .Ltmpxxx, .L__unnamed_xx, .LASANPC, etc. */ + ".L", + /* arm64 EFI stub namespace */ + "__efistub_", + /* arm64 local symbols in PIE namespace */ + "__pi_$", + "__pi_.L", + /* arm64 local symbols in non-VHE KVM namespace */ + "__kvm_nvhe_$", + "__kvm_nvhe_.L", + /* CFI type identifiers */ + "__kcfi_typeid_", + "__kvm_nvhe___kcfi_typeid_", + "__pi___kcfi_typeid_", + /* CRC from modversions */ + "__crc_", + /* EXPORT_SYMBOL (symbol name) */ + "__kstrtab_", + /* EXPORT_SYMBOL (namespace) */ + "__kstrtabns_", + /* MODULE_DEVICE_TABLE (symbol name) */ + "__mod_device_table__", +}; + +static const char *const sysmap_omit_suffixes[] = { + /* arm */ + "_from_arm", + "_from_thumb", + "_veneer", +}; + +static const char *const sysmap_omit_names[] = { + /* for LoongArch? */ + "L0", + /* ppc */ + "_SDA_BASE_", + "_SDA2_BASE_", +}; + +/* Symbols that contain the pattern anywhere are dropped. */ +static const char *const sysmap_omit_patterns[] = { + /* ppc stub */ + ".long_branch.", + ".plt_branch.", +}; + +/* __*Thunk_: the linker's range extension thunks on arm. */ +static bool is_range_thunk(const char *name) +{ + const char *p; + + if (!string_starts_with(name, "__")) + return false; + for (p = name + 2; isalnum((unsigned char)*p); p++) + ; + return p - name >= 7 && *p == '_' && strncmp(p - 5, "Thunk", 5) == 0; +} + +/* __UNIQUE_ID_modinfo_: the MODULE_INFO() strings of built-in code. */ +static bool is_modinfo_id(const char *name) +{ + static const char prefix[] = "__UNIQUE_ID_modinfo_"; + const char *p; + + if (!string_starts_with(name, prefix)) + return false; + for (p = name + strlen(prefix); isdigit((unsigned char)*p); p++) + ; + return !*p; +} + +static bool sysmap_omits(const char *name, char type) +{ + size_t i; + + /* Absolute, undefined and debugging symbols. */ + if (type == 'a' || type == 'N' || type == 'U' || type == 'w') + return true; + + for (i = 0; i < ARRAY_SIZE(sysmap_omit_prefixes); i++) + if (string_starts_with(name, sysmap_omit_prefixes[i])) + return true; + for (i = 0; i < ARRAY_SIZE(sysmap_omit_suffixes); i++) + if (string_ends_with(name, sysmap_omit_suffixes[i])) + return true; + for (i = 0; i < ARRAY_SIZE(sysmap_omit_names); i++) + if (strcmp(name, sysmap_omit_names[i]) == 0) + return true; + for (i = 0; i < ARRAY_SIZE(sysmap_omit_patterns); i++) + if (strstr(name, sysmap_omit_patterns[i])) + return true; + + return is_range_thunk(name) || is_modinfo_id(name); +} + +/* nm's letter for a symbol defined in a section, as BFD classifies it. */ +static char section_symbol_type(Elf_Shdr *shdr, const char *secname) +{ + static const char *const debug_prefixes[] = { + ".debug", ".zdebug", ".gnu.debuglto_.debug_", + ".gnu.linkonce.wi.", ".line", ".stab", + }; + const uint64_t flags = shdr_flags(shdr); + size_t i; + + if (flags & SHF_EXECINSTR) + return 't'; + if (flags & SHF_ALLOC) { + if (shdr_type(shdr) == SHT_NOBITS) + return 'b'; + return flags & SHF_WRITE ? 'd' : 'r'; + } + for (i = 0; i < ARRAY_SIZE(debug_prefixes); i++) + if (string_starts_with(secname, debug_prefixes[i])) + return 'N'; + if (shdr_type(shdr) != SHT_NOBITS && !(flags & SHF_WRITE)) + return 'n'; + return '?'; +} + +/* The letter nm prints for a symbol, or 0 for one it leaves out. */ +static char elf_symbol_type(const struct elf_file *elf, Elf_Sym *sym) +{ + unsigned int bind = sym_bind(sym), type = sym_type(sym); + unsigned int shndx = sym_shndx(sym); + Elf_Shdr *shdr; + char c; + + if (type == STT_SECTION || type == STT_FILE) + return 0; + if (shndx == SHN_COMMON) + return 'C'; + if (shndx == SHN_UNDEF) { + if (bind == STB_WEAK) + return type == STT_OBJECT ? 'v' : 'w'; + return 'U'; + } + if (type == STT_GNU_IFUNC) + return 'i'; + if (bind == STB_WEAK) + return type == STT_OBJECT ? 'V' : 'W'; + if (bind == STB_GNU_UNIQUE) + return 'u'; + if (bind != STB_GLOBAL && bind != STB_LOCAL) + return '?'; + + if (shndx == SHN_ABS) { + c = 'a'; + } else if (shndx < elf->shnum) { + shdr = elf_section(elf, shndx); + c = section_symbol_type(shdr, elf_section_name(elf, shdr)); + } else { + return '?'; + } + + return bind == STB_GLOBAL ? toupper(c) : c; +} + +/* nm -n order: by address, then by name. */ +static int compare_symbols(const void *a, const void *b) +{ + const struct sysmap_symbol *sa = a, *sb = b; + + if (sa->addr != sb->addr) + return sa->addr < sb->addr ? -1 : 1; + return strcmp(sa->name, sb->name); +} + +/* The symbols "nm -n | mksysmap" would list, in that order. */ +static struct sysmap_symbol *elf_read_symbols(const struct elf_file *elf, + size_t *nr_kept) +{ + struct sysmap_symbol *syms = xmalloc(elf->nr_syms * sizeof(*syms)); + size_t i, n = 0; + + for (i = 1; i < elf->nr_syms; i++) { + Elf_Sym *sym = elf_symbol(elf, i); + const char *name = elf->strtab + sym_name(sym); + char type = elf_symbol_type(elf, sym); + + if (!type || sysmap_omits(name, type)) + continue; + + syms[n].addr = sym_value(sym); + syms[n].name = name; + syms[n].type = type; + n++; + } + + qsort(syms, n, sizeof(*syms), compare_symbols); + *nr_kept = n; + return syms; +} + +struct sysmap *sysmap_read(const char *path) +{ + struct sysmap *map = xcalloc(1, sizeof(*map)); + + /* The names point into the mapping; keep it until the map is freed. */ + if (elf_open_ro(path, (1 << ET_EXEC) | (1 << ET_DYN), &map->elf)) + exit(EXIT_FAILURE); + map->syms = elf_read_symbols(&map->elf, &map->nr_syms); + map->addr_width = elf_map_long_size(map->elf.base) * 2; + + return map; +} + +void sysmap_write(const struct sysmap *map, FILE *out) +{ + size_t i; + + for (i = 0; i < map->nr_syms; i++) { + const struct sysmap_symbol *s = &map->syms[i]; + + fprintf(out, "%0*llx %c %s\n", map->addr_width, s->addr, s->type, + s->name); + } +} + +void sysmap_free(struct sysmap *map) +{ + free(map->syms); + elf_close(&map->elf); + free(map); +} diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 494852ade6d87a..ecb444078c1ca3 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -5,7 +5,15 @@ * This software may be used and distributed according to the terms * of the GNU General Public License, incorporated herein by reference. * - * Usage: kallsyms [--all-symbols] in.map > out.S + * Usage: kallsyms [--all-symbols] [--pc-relative] [--sysmap=out.map] in out.bin > out.S + * kallsyms --sysmap=out.map in + * + * in is vmlinux; an empty file stands for the first link, which has no + * symbols yet, and gives an empty table. --sysmap also writes the symbols + * in System.map format. + * + * The byte tables go to out.bin and are pulled into out.S with .incbin; + * wider tables stay assembler source for endianness and relocations. * * Table compression uses all the unused char codes on the symbols and * maps these to the most used substrings (tokens). For instance, it might @@ -18,7 +26,6 @@ * */ -#include #include #include #include @@ -26,10 +33,10 @@ #include #include #include - +#include #include -#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof(arr[0])) +#include "kallsyms.h" #define KSYM_NAME_LEN 512 @@ -58,20 +65,60 @@ static unsigned int table_size, table_cnt; static int all_symbols; static int pc_relative; -static int token_profit[0x10000]; +/* A dynamic array of symbols, encoded by symbol index. */ +struct sym_arr { + unsigned int *sym_indexes; + unsigned int cnt, cap; +}; + +/* Every pair of bytes is a token. */ +#define NR_TOKENS 0x10000 + +static int token_profit[NR_TOKENS]; +static struct sym_arr token_syms[NR_TOKENS]; /* the table that holds the result of the compression */ static unsigned char best_table[256][2]; static unsigned char best_table_len[256]; +static unsigned int sym_arr_last(const struct sym_arr *arr) +{ + return arr->cnt ? arr->sym_indexes[arr->cnt - 1] : UINT_MAX; +} + +static void sym_arr_maybe_expand(struct sym_arr *arr) +{ + if (arr->cap > arr->cnt) + return; + + arr->cap = arr->cap ? arr->cap * 2 : 16; + arr->sym_indexes = xrealloc(arr->sym_indexes, + arr->cap * sizeof(*arr->sym_indexes)); +} + +static void sym_arr_add(struct sym_arr *arr, unsigned int sym_idx) +{ + sym_arr_maybe_expand(arr); + arr->sym_indexes[arr->cnt++] = sym_idx; +} + +static void sym_arr_free(struct sym_arr *arr) +{ + free(arr->sym_indexes); + arr->sym_indexes = NULL; + arr->cnt = 0; + arr->cap = 0; +} static void usage(void) { - fprintf(stderr, "Usage: kallsyms [--all-symbols] in.map > out.S\n"); + fprintf(stderr, "Usage: kallsyms [--all-symbols] [--pc-relative] [--sysmap=out.map]\n" + " in out.bin > out.S\n" + " kallsyms --sysmap=out.map vmlinux\n"); exit(1); } -static char *sym_name(const struct sym_entry *s) +static char *sym_entry_name(const struct sym_entry *s) { return (char *)s->sym + 1; } @@ -112,37 +159,12 @@ static void check_symbol_range(const char *sym, unsigned long long addr, } } -static struct sym_entry *read_symbol(FILE *in, char **buf, size_t *buf_len) +static struct sym_entry *add_symbol(unsigned long long addr, char type, + const char *name) { - char *name, type, *p; - unsigned long long addr; - size_t len; - ssize_t readlen; + size_t len = strlen(name); struct sym_entry *sym; - errno = 0; - readlen = getline(buf, buf_len, in); - if (readlen < 0) { - if (errno) { - perror("read_symbol"); - exit(EXIT_FAILURE); - } - return NULL; - } - - if ((*buf)[readlen - 1] == '\n') - (*buf)[readlen - 1] = 0; - - addr = strtoull(*buf, &p, 16); - - if (*buf == p || *p++ != ' ' || !isascii((type = *p++)) || *p++ != ' ') { - fprintf(stderr, "line format error\n"); - exit(EXIT_FAILURE); - } - - name = p; - len = strlen(name); - if (len >= KSYM_NAME_LEN) { fprintf(stderr, "Symbol %s too long for kallsyms (%zu >= %d).\n" "Please increase KSYM_NAME_LEN both in kernel and kallsyms.c\n", @@ -167,7 +189,7 @@ static struct sym_entry *read_symbol(FILE *in, char **buf, size_t *buf_len) sym->addr = addr; sym->len = len; sym->sym[0] = type; - strcpy(sym_name(sym), name); + strcpy(sym_entry_name(sym), name); return sym; } @@ -188,14 +210,9 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } -static bool string_starts_with(const char *s, const char *prefix) -{ - return strncmp(s, prefix, strlen(prefix)) == 0; -} - static int symbol_valid(const struct sym_entry *s) { - const char *name = sym_name(s); + const char *name = sym_entry_name(s); /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ @@ -245,36 +262,55 @@ static void shrink_table(void) table_cnt = pos; } -static void read_map(const char *in) +static void add_table_entry(struct sym_entry *sym) { - FILE *fp; - struct sym_entry *sym; - char *buf = NULL; - size_t buflen = 0; + sym->seq = table_cnt; - fp = fopen(in, "r"); - if (!fp) { - perror(in); - exit(1); + if (table_cnt >= table_size) { + table_size = table_size ? table_size * 2 : 10000; + table = xrealloc(table, sizeof(*table) * table_size); } - while (!feof(fp)) { - sym = read_symbol(fp, &buf, &buflen); - if (!sym) - continue; + table[table_cnt++] = sym; +} - sym->seq = table_cnt; +static bool file_is_empty(const char *path) +{ + struct stat st; - if (table_cnt >= table_size) { - table_size += 10000; - table = xrealloc(table, sizeof(*table) * table_size); - } + if (stat(path, &st)) { + perror(path); + exit(EXIT_FAILURE); + } + + return st.st_size == 0; +} - table[table_cnt++] = sym; +/* + * Read the symbols from vmlinux, writing System.map if asked to. The first + * link has no symbols yet: an empty file gives an empty table. + */ +static void read_elf(const char *path, FILE *sysmap_out) +{ + struct sysmap *map; + size_t i; + + if (file_is_empty(path)) + return; + + map = sysmap_read(path); + if (sysmap_out) + sysmap_write(map, sysmap_out); + + for (i = 0; i < map->nr_syms; i++) { + const struct sysmap_symbol *s = &map->syms[i]; + struct sym_entry *sym = add_symbol(s->addr, s->type, s->name); + + if (sym) + add_table_entry(sym); } - free(buf); - fclose(fp); + sysmap_free(map); } static void output_label(const char *label) @@ -284,6 +320,26 @@ static void output_label(const char *label) printf("%s:\n", label); } +static long bin_pos(FILE *file) +{ + const long pos = ftell(file); + + if (pos < 0) { + perror("kallsyms: ftell"); + exit(EXIT_FAILURE); + } + + return pos; +} + +static void write_incbin(const char *filename, long start, long end) +{ + if (start >= end) + return; + + printf("\t.incbin \"%s\", %ld, %ld\n", filename, start, end - start); +} + /* uncompress a compressed symbol. When this function is called, the best table * might still be compressed itself, so the function needs to be recursive */ static int expand_symbol(const unsigned char *data, int len, char *result) @@ -317,7 +373,7 @@ static int compare_names(const void *a, const void *b) const struct sym_entry *sa = *(const struct sym_entry **)a; const struct sym_entry *sb = *(const struct sym_entry **)b; - ret = strcmp(sym_name(sa), sym_name(sb)); + ret = strcmp(sym_entry_name(sa), sym_entry_name(sb)); if (!ret) { if (sa->addr > sb->addr) return 1; @@ -336,11 +392,12 @@ static void sort_symbols_by_name(void) qsort(table, table_cnt, sizeof(table[0]), compare_names); } -static void write_src(void) +static void write_src(FILE *out_bin_file, const char *out_bin_name) { - unsigned int i, k, off; + unsigned int i, off; unsigned int best_idx[256]; unsigned int *markers, markers_cnt; + long bin_start; char buf[KSYM_NAME_LEN]; printf("\t.section .rodata, \"a\"\n"); @@ -355,6 +412,7 @@ static void write_src(void) markers = xmalloc(sizeof(*markers) * markers_cnt); output_label("kallsyms_names"); + bin_start = bin_pos(out_bin_file); off = 0; for (i = 0; i < table_cnt; i++) { if ((i & 0xFF) == 0) @@ -378,26 +436,24 @@ static void write_src(void) /* Encode length with ULEB128. */ if (table[i]->len <= 0x7F) { /* Most symbols use a single byte for the length. */ - printf("\t.byte 0x%02x", table[i]->len); + fputc(table[i]->len, out_bin_file); off += table[i]->len + 1; } else { /* "Big" symbols use two bytes. */ - printf("\t.byte 0x%02x, 0x%02x", - (table[i]->len & 0x7F) | 0x80, - (table[i]->len >> 7) & 0x7F); + fputc((table[i]->len & 0x7F) | 0x80, out_bin_file); + fputc((table[i]->len >> 7) & 0x7F, out_bin_file); off += table[i]->len + 2; } - for (k = 0; k < table[i]->len; k++) - printf(", 0x%02x", table[i]->sym[k]); + fwrite(table[i]->sym, 1, table[i]->len, out_bin_file); /* * Now that we wrote out the compressed symbol name, restore the - * original name and print it in the comment. + * original name for the comments below. */ expand_symbol(table[i]->sym, table[i]->len, buf); strcpy((char *)table[i]->sym, buf); - printf("\t/* %s */\n", table[i]->sym); } + write_incbin(out_bin_name, bin_start, bin_pos(out_bin_file)); printf(".size kallsyms_names, . - kallsyms_names\n"); printf("\n"); @@ -410,13 +466,15 @@ static void write_src(void) free(markers); output_label("kallsyms_token_table"); + bin_start = bin_pos(out_bin_file); off = 0; for (i = 0; i < 256; i++) { best_idx[i] = off; expand_symbol(best_table[i], best_table_len[i], buf); - printf("\t.asciz\t\"%s\"\n", buf); + fwrite(buf, 1, strlen(buf) + 1, out_bin_file); off += strlen(buf) + 1; } + write_incbin(out_bin_name, bin_start, bin_pos(out_bin_file)); printf(".size kallsyms_token_table, . - kallsyms_token_table\n"); printf("\n"); @@ -449,15 +507,25 @@ static void write_src(void) sort_symbols_by_name(); output_label("kallsyms_seqs_of_names"); - for (i = 0; i < table_cnt; i++) - printf("\t.byte 0x%02x, 0x%02x, 0x%02x\t/* %s */\n", - (unsigned char)(table[i]->seq >> 16), - (unsigned char)(table[i]->seq >> 8), - (unsigned char)(table[i]->seq >> 0), - table[i]->sym); + bin_start = bin_pos(out_bin_file); + for (i = 0; i < table_cnt; i++) { + fputc(table[i]->seq >> 16, out_bin_file); + fputc(table[i]->seq >> 8, out_bin_file); + fputc(table[i]->seq >> 0, out_bin_file); + } + write_incbin(out_bin_name, bin_start, bin_pos(out_bin_file)); printf("\n"); } +static unsigned int token_index(unsigned char first, unsigned char second) +{ + return first + (second << 8); +} + +static unsigned int sym_token_index(const unsigned char *symbol, int first_idx) +{ + return token_index(symbol[first_idx], symbol[first_idx + 1]); +} /* table lookup compression functions */ @@ -467,7 +535,7 @@ static void learn_symbol(const unsigned char *symbol, int len) int i; for (i = 0; i < len - 1; i++) - token_profit[ symbol[i] + (symbol[i + 1] << 8) ]++; + token_profit[sym_token_index(symbol, i)]++; } /* decrease the count for all the possible tokens in a symbol */ @@ -476,16 +544,76 @@ static void forget_symbol(const unsigned char *symbol, int len) int i; for (i = 0; i < len - 1; i++) - token_profit[ symbol[i] + (symbol[i + 1] << 8) ]--; + token_profit[sym_token_index(symbol, i)]--; +} + +static void token_add_symbol(unsigned int token_idx, unsigned int sym_idx) +{ + struct sym_arr *arr = &token_syms[token_idx]; + + /* Symbol indexes kept in sorted order, check for duplicate. */ + if (sym_arr_last(arr) == sym_idx) + return; + + sym_arr_add(arr, sym_idx); +} + +static void symbol_index_all_tokens(const unsigned char *symbol, int len, + unsigned int sym_idx) +{ + int i; + + for (i = 0; i < len - 1; i++) { + const unsigned int token_idx = sym_token_index(symbol, i); + + token_add_symbol(token_idx, sym_idx); + } +} + +/* + * The symbol just got compressed. The only parts of the symbol that changed + * meaningfully are those containing the newly assigned compressed char, so + * index those. + */ +static void symbol_index_new_tokens(const unsigned char *symbol, int len, + unsigned int sym_idx, int compressed_chr) +{ + int i; + + for (i = 0; i < len - 1; i++) { + const unsigned int token_idx = sym_token_index(symbol, i); + + if (symbol[i] == compressed_chr || + symbol[i + 1] == compressed_chr) + token_add_symbol(token_idx, sym_idx); + } } -/* do the initial token count */ static void build_initial_token_table(void) { unsigned int i; for (i = 0; i < table_cnt; i++) learn_symbol(table[i]->sym, table[i]->len); + + /* + * The initial occurrence counts tell us exactly how much memory should + * be reserved for each token's symbol array. + */ + for (i = 0; i < ARRAY_SIZE(token_syms); i++) { + const int nr_syms = token_profit[i]; + + if (!nr_syms) + continue; + + token_syms[i].cap = nr_syms; + token_syms[i].sym_indexes = + xmalloc(nr_syms * sizeof(unsigned int)); + } + + /* For every symbol, index every token -> symbol it is present in. */ + for (i = 0; i < table_cnt; i++) + symbol_index_all_tokens(table[i]->sym, table[i]->len, i); } static unsigned char *find_token(unsigned char *str, int len, @@ -502,27 +630,31 @@ static unsigned char *find_token(unsigned char *str, int len, /* replace a given token in all the valid symbols. Use the sampled symbols * to update the counts */ -static void compress_symbols(const unsigned char *str, int idx) +static void compress_symbols(const unsigned char *str, int compressed_chr) { - unsigned int i, len, size; + const unsigned int token_idx = sym_token_index(str, 0); + struct sym_arr *arr = &token_syms[token_idx]; + unsigned int sym_idx, j, len, size; unsigned char *p1, *p2; - for (i = 0; i < table_cnt; i++) { + /* Iterate through all symbols this token is found in and compress. */ + for (j = 0; j < arr->cnt; j++) { + sym_idx = arr->sym_indexes[j]; - len = table[i]->len; - p1 = table[i]->sym; + len = table[sym_idx]->len; + p1 = table[sym_idx]->sym; /* find the token on the symbol */ p2 = find_token(p1, len, str); if (!p2) continue; /* decrease the counts for this symbol's tokens */ - forget_symbol(table[i]->sym, len); + forget_symbol(table[sym_idx]->sym, len); size = len; do { - *p2 = idx; + *p2 = compressed_chr; p2++; size -= (p2 - p1); memmove(p2, p2 + 1, size); @@ -536,11 +668,15 @@ static void compress_symbols(const unsigned char *str, int idx) } while (p2); - table[i]->len = len; + table[sym_idx]->len = len; /* increase the counts for this symbol's new tokens */ - learn_symbol(table[i]->sym, len); + learn_symbol(table[sym_idx]->sym, len); + symbol_index_new_tokens(table[sym_idx]->sym, len, sym_idx, + compressed_chr); } + + sym_arr_free(arr); /* No symbol contains this token any more. */ } /* search the token with the maximum profit */ @@ -551,7 +687,7 @@ static int find_best_token(void) bestprofit=-10000; best = 0; - for (i = 0; i < 0x10000; i++) { + for (i = 0; i < ARRAY_SIZE(token_profit); i++) { if (token_profit[i] > bestprofit) { best = i; bestprofit = token_profit[i]; @@ -615,7 +751,7 @@ static void optimize_token_table(void) /* guess for "linker script provide" symbol */ static int may_be_linker_script_provide_symbol(const struct sym_entry *se) { - const char *symbol = sym_name(se); + const char *symbol = sym_entry_name(se); int len = se->len - 1; if (len < 8) @@ -672,8 +808,8 @@ static int compare_symbols(const void *a, const void *b) return wa - wb; /* sort by the number of prefix underscores */ - wa = strspn(sym_name(sa), "_"); - wb = strspn(sym_name(sb), "_"); + wa = strspn(sym_entry_name(sa), "_"); + wb = strspn(sym_entry_name(sb), "_"); if (wa != wb) return wa - wb; @@ -688,10 +824,14 @@ static void sort_symbols(void) int main(int argc, char **argv) { + const char *in, *sysmap = NULL, *out_bin_name; + FILE *sysmap_out = NULL, *out_bin_file; + while (1) { static const struct option long_options[] = { {"all-symbols", no_argument, &all_symbols, 1}, {"pc-relative", no_argument, &pc_relative, 1}, + {"sysmap", required_argument, NULL, 's'}, {}, }; @@ -699,18 +839,55 @@ int main(int argc, char **argv) if (c == -1) break; - if (c != 0) + if (c == 's') + sysmap = optarg; + else if (c != 0) usage(); } - if (optind >= argc) + if (optind + 2 != argc && !(sysmap && optind + 1 == argc)) usage(); - read_map(argv[optind]); + in = argv[optind]; + if (sysmap) { + sysmap_out = fopen(sysmap, "w"); + if (!sysmap_out) { + perror(sysmap); + exit(EXIT_FAILURE); + } + } + + if (optind + 1 == argc) { + read_elf(in, sysmap_out); + if (ferror(sysmap_out) || fclose(sysmap_out)) { + perror(sysmap); + exit(EXIT_FAILURE); + } + return 0; + } + + out_bin_name = argv[optind + 1]; + out_bin_file = fopen(out_bin_name, "w"); + if (!out_bin_file) { + perror(out_bin_name); + exit(EXIT_FAILURE); + } + + read_elf(in, sysmap_out); + if (sysmap_out && (ferror(sysmap_out) || fclose(sysmap_out))) { + perror(sysmap); + exit(EXIT_FAILURE); + } shrink_table(); sort_symbols(); optimize_token_table(); - write_src(); + write_src(out_bin_file, out_bin_name); + + /* Write errors are latched in the stream, so one check at the end covers them all. */ + if (ferror(out_bin_file) || fclose(out_bin_file)) { + perror(out_bin_name); + exit(EXIT_FAILURE); + } return 0; } diff --git a/scripts/kallsyms.h b/scripts/kallsyms.h new file mode 100644 index 00000000000000..4b2801ba628593 --- /dev/null +++ b/scripts/kallsyms.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef KALLSYMS_H +#define KALLSYMS_H + +#include +#include +#include +#include + +#include +#include "elf-parse.h" + +static inline bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + +static inline bool string_ends_with(const char *s, const char *suffix) +{ + size_t len = strlen(s), suffix_len = strlen(suffix); + + return len >= suffix_len && strcmp(s + len - suffix_len, suffix) == 0; +} + +/* A symbol as nm lists it. */ +struct sysmap_symbol { + unsigned long long addr; + const char *name; + char type; +}; + +/* The symbols of an ELF file that System.map lists, in its order. */ +struct sysmap { + struct sysmap_symbol *syms; + size_t nr_syms; + int addr_width; /* hex digits of an address */ + struct elf_file elf; /* the mapping the names point into */ +}; + +struct sysmap *sysmap_read(const char *path); +void sysmap_write(const struct sysmap *map, FILE *out); +void sysmap_free(struct sysmap *map); + +#endif /* KALLSYMS_H */ diff --git a/scripts/kconfig/conf.c b/scripts/kconfig/conf.c index fe8ba09b003910..31c6fd25e6ecd8 100644 --- a/scripts/kconfig/conf.c +++ b/scripts/kconfig/conf.c @@ -186,12 +186,23 @@ static void conf_set_all_new_symbols(enum conf_def_mode mode) if (mode == def_random) { int n, p[3]; + bool warned = false; char *env = getenv("KCONFIG_PROBABILITY"); n = 0; while (env && *env) { char *endp; - int tmp = strtol(env, &endp, 10); + long tmp = strtol(env, &endp, 10); + + if (!isdigit((unsigned char)*env) || + (*endp && *endp != ':') || + (*endp == ':' && (!endp[1] || n == 2))) { + if (!warned) { + fprintf(stderr, + "warning: KCONFIG_PROBABILITY has malformed format\n"); + warned = true; + } + } if (tmp >= 0 && tmp <= 100) { p[n++] = tmp; @@ -227,6 +238,9 @@ static void conf_set_all_new_symbols(enum conf_def_mode mode) perror("KCONFIG_PROBABILITY"); exit(1); } + + if (warned && getenv("KCONFIG_WERROR")) + exit(1); } menu_for_each_entry(menu) { @@ -469,6 +483,9 @@ static void conf_choice(struct menu *menu) } printf("\n"); } + /* Nothing to ask if none of the choice values is visible. */ + if (!cnt) + return; printf("%*schoice", indent - 1, ""); if (cnt == 1) { printf("[1]: 1\n"); diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c index 4234a51d16fd18..2227d89d6328f8 100644 --- a/scripts/kconfig/confdata.c +++ b/scripts/kconfig/confdata.c @@ -354,6 +354,12 @@ static int conf_set_sym_val(struct symbol *sym, int def, int def_flags, char *p) case S_INT: case S_HEX: if (sym_string_valid(sym, p)) { + if (def != S_DEF_AUTO && + !sym_string_check_bounds(sym, p)) + /* hex uses 64-bit unsigned integer */ + conf_warning("value '%s' for %s is outside the 64-bit %s integer bounds", + p, sym->name, + sym->type == S_INT ? "signed" : "unsigned"); sym->def[def].val = xstrdup(p); sym->flags |= def_flags; } else { diff --git a/scripts/kconfig/kconfig-sym-check.pl b/scripts/kconfig/kconfig-sym-check.pl index daa5285fdefc85..c8dd07f8b27c58 100755 --- a/scripts/kconfig/kconfig-sym-check.pl +++ b/scripts/kconfig/kconfig-sym-check.pl @@ -90,7 +90,7 @@ sub indent_depth { next; } - if (/^\s*(default|def_bool|def_tristate|select|depends\s+on|imply|visible\s+if|range|if|bool|tristate|int|hex|string|prompt)\s+(.+)\s*$/) { + if (/^\s*(default|def_bool|def_tristate|def_string|def_int|def_hex|select|depends\s+on|imply|visible\s+if|range|if|bool|tristate|int|hex|string|prompt)\s+(.+)\s*$/) { my $s = $2; $s =~ s/"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'//g; $s =~ s/#.*//; diff --git a/scripts/kconfig/lexer.l b/scripts/kconfig/lexer.l index a6155422b4a688..1fa521199d4a30 100644 --- a/scripts/kconfig/lexer.l +++ b/scripts/kconfig/lexer.l @@ -105,6 +105,9 @@ n [A-Za-z0-9_-] "comment" return T_COMMENT; "config" return T_CONFIG; "def_bool" return T_DEF_BOOL; +"def_hex" return T_DEF_HEX; +"def_int" return T_DEF_INT; +"def_string" return T_DEF_STRING; "def_tristate" return T_DEF_TRISTATE; "default" return T_DEFAULT; "depends" return T_DEPENDS; diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h index 7e6f6ca299cf1a..bbc99f75b416da 100644 --- a/scripts/kconfig/lkc.h +++ b/scripts/kconfig/lkc.h @@ -89,7 +89,7 @@ struct property *menu_add_prompt(enum prop_type type, const char *prompt, struct expr *dep); void menu_add_expr(enum prop_type type, struct expr *expr, struct expr *dep); void menu_add_symbol(enum prop_type type, struct symbol *sym, struct expr *dep); -void menu_finalize(void); +int menu_finalize(void); void menu_set_type(int type); extern struct menu rootmenu; diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h index 8914b4e8f2a811..8b436c87ba4ae3 100644 --- a/scripts/kconfig/lkc_proto.h +++ b/scripts/kconfig/lkc_proto.h @@ -31,6 +31,7 @@ bool sym_set_tristate_value(struct symbol *sym,tristate tri); void choice_set_value(struct menu *choice, struct symbol *sym); tristate sym_toggle_tristate_value(struct symbol *sym); bool sym_string_valid(struct symbol *sym, const char *newval); +bool sym_string_check_bounds(struct symbol *sym, const char *str); bool sym_string_within_range(struct symbol *sym, const char *str); bool sym_set_string_value(struct symbol *sym, const char *newval); bool sym_is_changeable(const struct symbol *sym); diff --git a/scripts/kconfig/lxdialog/textbox.c b/scripts/kconfig/lxdialog/textbox.c index 0abaf635978f98..3fe7682fc8a8a8 100644 --- a/scripts/kconfig/lxdialog/textbox.c +++ b/scripts/kconfig/lxdialog/textbox.c @@ -145,8 +145,6 @@ static void refresh_text_box(WINDOW *dialog, WINDOW *box, int boxh, int boxw, /* * Display text from a file in a dialog box. - * - * keys is a null-terminated array */ int dialog_textbox(const char *title, const char *tbuf, int initial_height, int initial_width, int *_vscroll, int *_hscroll, diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c index 9c079e92a9ed62..bda3e26502161f 100644 --- a/scripts/kconfig/menu.c +++ b/scripts/kconfig/menu.c @@ -234,17 +234,42 @@ void menu_add_symbol(enum prop_type type, struct symbol *sym, struct expr *dep) menu_add_prop(type, expr_alloc_symbol(sym), dep); } -static int menu_validate_number(struct symbol *sym, struct symbol *sym2) -{ - return sym2->type == S_INT || sym2->type == S_HEX || - (sym2->type == S_UNKNOWN && sym_string_valid(sym, sym2->name)); +/* Validate the sym2 value for numeric sym. */ +static int menu_validate_number(struct symbol *sym, struct symbol *sym2, + const struct property *prop) +{ + if (sym->type != S_INT && sym->type != S_HEX) + return 0; + + if (sym2->type == sym->type) + return 0; + + if (sym2->type != S_UNKNOWN || + !sym_string_valid(sym, sym2->name)) { + fprintf(stderr, "%s:%d: error: '%s' is an invalid value for '%s'\n", + prop->filename, prop->lineno, sym2->name, + sym_type_name(sym->type)); + return 1; + } + + if (!sym_string_check_bounds(sym, sym2->name)) { + fprintf(stderr, + "%s:%d: error: %s constant '%s' is outside the 64-bit %s bounds\n", + prop->filename, prop->lineno, sym_type_name(sym->type), + sym2->name, sym->type == S_INT ? "signed" : "unsigned"); + + return 1; + } + + return 0; } -static void sym_check_prop(struct symbol *sym) +static int sym_check_prop(struct symbol *sym) { struct property *prop; struct symbol *sym2; char *use; + int errors = 0; for (prop = sym->prop; prop; prop = prop->next) { switch (prop->type) { @@ -258,10 +283,7 @@ static void sym_check_prop(struct symbol *sym) break; sym2 = prop_get_symbol(prop); if (sym->type == S_HEX || sym->type == S_INT) { - if (!menu_validate_number(sym, sym2)) - prop_warn(prop, - "'%s': number is invalid", - sym->name); + errors += menu_validate_number(sym, sym2, prop); } if (sym_is_choice(sym)) { struct menu *choice = sym_get_choice_menu(sym2); @@ -287,27 +309,33 @@ static void sym_check_prop(struct symbol *sym) "'%s' has wrong type. '%s' only " "accept arguments of bool and " "tristate type", sym2->name, use); + if (sym_is_choice_value(sym2)) + prop_warn(prop, + "config symbol '%s' uses %s for '%s', but '%s' is a choice value", + sym->name, use, sym2->name, sym2->name); break; case P_RANGE: if (sym->type != S_INT && sym->type != S_HEX) prop_warn(prop, "range is only allowed " "for int or hex symbols"); - if (!menu_validate_number(sym, prop->expr->left.sym) || - !menu_validate_number(sym, prop->expr->right.sym)) - prop_warn(prop, "range is invalid"); + errors += menu_validate_number(sym, prop->expr->left.sym, prop); + errors += menu_validate_number(sym, prop->expr->right.sym, prop); break; default: ; } } + + return errors; } -static void _menu_finalize(struct menu *parent, bool inside_choice) +static int _menu_finalize(struct menu *parent, bool inside_choice) { struct menu *menu, *last_menu; struct symbol *sym; struct property *prop; struct expr *basedep, *dep, *dep2; + int errors = 0; sym = parent->sym; if (parent->list) { @@ -393,7 +421,7 @@ static void _menu_finalize(struct menu *parent, bool inside_choice) * moving on */ for (menu = parent->list; menu; menu = menu->next) - _menu_finalize(menu, sym && sym_is_choice(sym)); + errors += _menu_finalize(menu, sym && sym_is_choice(sym)); } else if (!inside_choice && sym) { /* * Automatic submenu creation. If sym is a symbol and A, B, C, @@ -461,7 +489,7 @@ static void _menu_finalize(struct menu *parent, bool inside_choice) } /* Superset, put in submenu */ next: - _menu_finalize(menu, false); + errors += _menu_finalize(menu, false); menu->parent = parent; last_menu = menu; } @@ -519,14 +547,16 @@ static void _menu_finalize(struct menu *parent, bool inside_choice) menu_warn(parent, "config symbol defined without type"); /* Check properties connected to this symbol */ - sym_check_prop(sym); + errors += sym_check_prop(sym); sym->flags |= SYMBOL_WARNED; } + + return errors; } -void menu_finalize(void) +int menu_finalize(void) { - _menu_finalize(&rootmenu, false); + return _menu_finalize(&rootmenu, false); } bool menu_has_prompt(const struct menu *menu) diff --git a/scripts/kconfig/parser.y b/scripts/kconfig/parser.y index 5fb6f07b6ad292..ecb18c213d83b0 100644 --- a/scripts/kconfig/parser.y +++ b/scripts/kconfig/parser.y @@ -53,6 +53,9 @@ struct menu *current_menu, *current_entry, *current_choice; %token T_CONFIG %token T_DEFAULT %token T_DEF_BOOL +%token T_DEF_HEX +%token T_DEF_INT +%token T_DEF_STRING %token T_DEF_TRISTATE %token T_DEPENDS %token T_ENDCHOICE @@ -309,6 +312,9 @@ type: default: T_DEFAULT { $$ = S_UNKNOWN; } | T_DEF_BOOL { $$ = S_BOOLEAN; } + | T_DEF_HEX { $$ = S_HEX; } + | T_DEF_INT { $$ = S_INT; } + | T_DEF_STRING { $$ = S_STRING; } | T_DEF_TRISTATE { $$ = S_TRISTATE; } /* if entry */ @@ -587,7 +593,7 @@ void conf_parse(const char *name) menu_add_prompt(P_MENU, "Main menu", NULL); } - menu_finalize(); + yynerrs += menu_finalize(); menu_for_each_entry(menu) { struct menu *child; diff --git a/scripts/kconfig/streamline_config.pl b/scripts/kconfig/streamline_config.pl index 8677d1ca06a726..6a52a98eb40e41 100755 --- a/scripts/kconfig/streamline_config.pl +++ b/scripts/kconfig/streamline_config.pl @@ -145,6 +145,7 @@ sub read_config { my %objects; my %config2kfile; my %defaults; +my %types; my $var; my $iflevel = 0; my @ifdeps; @@ -224,6 +225,7 @@ sub read_kconfig { } elsif ($state ne "NONE" && /^\s*def(_(bool|tristate)|ault)\s+(\S.*)$/) { my $dep = $3; $defaults{$config} = 1; + $types{$config} = $2 if (defined($2)); if ($dep !~ /^\s*(y|m|n)\s*$/) { $dep =~ s/.*\sif\s+//; $depends{$config} .= " " . $dep; @@ -240,7 +242,13 @@ sub read_kconfig { } # configs without prompts must be selected - } elsif ($state ne "NONE" && /^\s*(tristate\s+\S|prompt\b)/) { + } elsif ($state ne "NONE" && /^\s*(bool|boolean)(\s+\S.*)?$/) { + $types{$config} = "bool"; + $prompts{$config} = 1 if (defined($2)); + } elsif ($state ne "NONE" && /^\s*tristate(\s+\S.*)?$/) { + $types{$config} = "tristate"; + $prompts{$config} = 1 if (defined($1)); + } elsif ($state ne "NONE" && /^\s*prompt\b/) { # note if the config has a prompt $prompts{$config} = 1; @@ -707,7 +715,15 @@ sub in_preserved_kconfigs { } print STDERR "module $module did not have configs"; foreach my $conf (@arr) { - print STDERR " " , $conf; + my $c = $conf; + $c =~ s/^CONFIG_//; + if (!$localyesconfig && defined($types{$c}) && $types{$c} eq "bool") { + print STDERR " $conf (boolean, not supported with =m)"; + } elsif (defined($types{$c})) { + print STDERR " $conf ($types{$c})"; + } else { + print STDERR " " , $conf; + } } print STDERR "\n"; } diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c index 7e81b3676ee944..4ced4dd6485ba4 100644 --- a/scripts/kconfig/symbol.c +++ b/scripts/kconfig/symbol.c @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -141,8 +142,15 @@ struct property *sym_get_range_prop(struct symbol *sym) return NULL; } -static long long sym_get_range_val(struct symbol *sym, int base) +union range_value { + long long s; + unsigned long long u; +}; + +static union range_value sym_get_range_val(struct symbol *sym, int base) { + union range_value val; + sym_calc_value(sym); switch (sym->type) { case S_INT: @@ -154,15 +162,25 @@ static long long sym_get_range_val(struct symbol *sym, int base) default: break; } - return strtoll(sym->curr.val, NULL, base); + if (base == 10) + val.s = strtoll(sym->curr.val, NULL, base); + else + /* HEX */ + val.u = strtoull(sym->curr.val, NULL, base); + return val; } -static void sym_validate_range(struct symbol *sym) +/* + * Return the nearest range bound for an out-of-range default value. + * Return NULL if the value is valid or the symbol has no active range. + */ +static struct symbol *sym_get_near_range_bound(struct symbol *sym, + const char *value) { struct property *prop; struct symbol *range_sym; int base; - long long val, val2; + union range_value val, val2; switch (sym->type) { case S_INT: @@ -172,21 +190,40 @@ static void sym_validate_range(struct symbol *sym) base = 16; break; default: - return; + return NULL; } prop = sym_get_range_prop(sym); if (!prop) - return; - val = strtoll(sym->curr.val, NULL, base); + return NULL; + if (base == 10) + val.s = strtoll(value, NULL, base); + else + val.u = strtoull(value, NULL, base); range_sym = prop->expr->left.sym; val2 = sym_get_range_val(range_sym, base); - if (val >= val2) { + + if (base == 10 && val.s >= val2.s) { range_sym = prop->expr->right.sym; val2 = sym_get_range_val(range_sym, base); - if (val <= val2) - return; + if (val.s <= val2.s) + return NULL; + } else if (base == 16 && val.u >= val2.u) { + range_sym = prop->expr->right.sym; + val2 = sym_get_range_val(range_sym, base); + if (val.u <= val2.u) + return NULL; } - sym->curr.val = range_sym->curr.val; + + return range_sym; +} + +static void sym_validate_range(struct symbol *sym) +{ + struct symbol *range_sym; + + range_sym = sym_get_near_range_bound(sym, sym->curr.val); + if (range_sym) + sym->curr.val = range_sym->curr.val; } static void sym_set_changed(struct symbol *sym) @@ -711,10 +748,26 @@ bool sym_string_valid(struct symbol *sym, const char *str) } } +bool sym_string_check_bounds(struct symbol *sym, const char *str) +{ + errno = 0; + + if (sym->type == S_INT) + strtoll(str, NULL, 10); + else if (sym->type == S_HEX) + strtoull(str, NULL, 16); + else + /* string */ + return true; + + return errno != ERANGE; +} + bool sym_string_within_range(struct symbol *sym, const char *str) { struct property *prop; long long val; + unsigned long long uval; switch (sym->type) { case S_STRING: @@ -722,21 +775,25 @@ bool sym_string_within_range(struct symbol *sym, const char *str) case S_INT: if (!sym_string_valid(sym, str)) return false; + if (!sym_string_check_bounds(sym, str)) + return false; prop = sym_get_range_prop(sym); if (!prop) return true; val = strtoll(str, NULL, 10); - return val >= sym_get_range_val(prop->expr->left.sym, 10) && - val <= sym_get_range_val(prop->expr->right.sym, 10); + return val >= sym_get_range_val(prop->expr->left.sym, 10).s && + val <= sym_get_range_val(prop->expr->right.sym, 10).s; case S_HEX: if (!sym_string_valid(sym, str)) return false; + if (!sym_string_check_bounds(sym, str)) + return false; prop = sym_get_range_prop(sym); if (!prop) return true; - val = strtoll(str, NULL, 16); - return val >= sym_get_range_val(prop->expr->left.sym, 16) && - val <= sym_get_range_val(prop->expr->right.sym, 16); + uval = strtoull(str, NULL, 16); + return uval >= sym_get_range_val(prop->expr->left.sym, 16).u && + uval <= sym_get_range_val(prop->expr->right.sym, 16).u; case S_BOOLEAN: case S_TRISTATE: switch (str[0]) { @@ -812,7 +869,7 @@ bool sym_set_string_value(struct symbol *sym, const char *newval) const char *sym_get_string_default(struct symbol *sym) { struct property *prop; - struct symbol *ds; + struct symbol *ds, *range_sym; const char *str = ""; tristate val; @@ -830,11 +887,6 @@ const char *sym_get_string_default(struct symbol *sym) val = EXPR_AND(expr_calc_value(prop->expr), prop->visible.tri); break; default: - /* - * The following fails to handle the situation - * where a default value is further limited by - * the valid range. - */ ds = prop_get_symbol(prop); if (ds != NULL) { sym_calc_value(ds); @@ -878,6 +930,11 @@ const char *sym_get_string_default(struct symbol *sym) default: break; } + + range_sym = sym_get_near_range_bound(sym, str); + if (range_sym) + str = range_sym->curr.val; + return str; } @@ -1196,6 +1253,7 @@ static struct symbol *sym_check_expr_deps(const struct expr *e) case E_GTH: case E_LEQ: case E_LTH: + case E_RANGE: case E_UNEQUAL: sym = sym_check_deps(e->left.sym); if (sym) @@ -1243,7 +1301,9 @@ static struct symbol *sym_check_sym_deps(struct symbol *sym) sym2 = sym_check_expr_deps(prop->visible.expr); if (sym2) break; - if (prop->type != P_DEFAULT || sym_is_choice(sym)) + if (sym_is_choice(sym)) + continue; + if (prop->type != P_DEFAULT && prop->type != P_RANGE) continue; stack.expr = &prop->expr; sym2 = sym_check_expr_deps(prop->expr); diff --git a/scripts/kconfig/tests/conftest.py b/scripts/kconfig/tests/conftest.py index 66f95e4ed58c19..d22c080a4e4065 100644 --- a/scripts/kconfig/tests/conftest.py +++ b/scripts/kconfig/tests/conftest.py @@ -57,9 +57,12 @@ def _run_conf(self, mode, dot_config=None, out_file='.config', # Override 'srctree' environment to make the test as the top directory extra_env['srctree'] = self._test_dir - # Clear KCONFIG_DEFCONFIG_LIST to keep unit tests from being affected - # by the user's environment. - extra_env['KCONFIG_DEFCONFIG_LIST'] = '' + # Clear all KCONFIG_* environment variables + env = { + varname: os.environ[varname] + for varname in os.environ.keys() + if not varname.startswith("KCONFIG_") + } # Run Kconfig in a temporary directory. # This directory is automatically removed when done. @@ -75,7 +78,7 @@ def _run_conf(self, mode, dot_config=None, out_file='.config', stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=temp_dir, - env=dict(os.environ, **extra_env)) + env=dict(env, **extra_env)) # If input key sequence is given, feed it to stdin. if in_keys: @@ -156,13 +159,13 @@ def oldconfig(self, dot_config=None, in_keys=None): return self._run_conf('--oldconfig', dot_config=dot_config, interactive=True, in_keys=in_keys) - def olddefconfig(self, dot_config=None): + def olddefconfig(self, dot_config=None, **kw): """Run olddefconfig. dot_config: .config file to use for configuration base (optional) returncode: exit status of the Kconfig executable """ - return self._run_conf('--olddefconfig', dot_config=dot_config) + return self._run_conf('--olddefconfig', dot_config=dot_config, **kw) def defconfig(self, defconfig): """Run defconfig. @@ -226,13 +229,15 @@ def randconfig(self, all_config=None, seed=None): return self._allconfig('rand', all_config, extra_env=extra_env) - def savedefconfig(self, dot_config): + def savedefconfig(self, dot_config, out_file='defconfig', **kw): """Run savedefconfig. dot_config: .config file for input + out_file: defconfig file for output returncode: exit status of the Kconfig executable """ - return self._run_conf('--savedefconfig', out_file='defconfig') + return self._run_conf('--savedefconfig={}'.format(out_file), + dot_config=dot_config, out_file=out_file, **kw) def listnewconfig(self, dot_config=None): """Run listnewconfig. @@ -312,6 +317,12 @@ def stderr_matches(self, expected): return self._matches('stderr', expected) +@pytest.fixture(autouse=True) +def clear_werror(monkeypatch): + # extra_env can set strict mode, but cannot remove an inherited flag. + monkeypatch.delenv('KCONFIG_WERROR', raising=False) + + @pytest.fixture(scope="module") def conf(request): """Create a Conf instance and provide it to test functions.""" diff --git a/scripts/kconfig/tests/def_type/Kconfig b/scripts/kconfig/tests/def_type/Kconfig new file mode 100644 index 00000000000000..fbee37a631790f --- /dev/null +++ b/scripts/kconfig/tests/def_type/Kconfig @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 +# The def_ shorthands: a type definition plus a default value. + +config MODULES + bool "Enable loadable module support" + modules + default y + +config GUARD + bool "Guard symbol" + +config DEF_BOOL + def_bool GUARD + +config DEF_TRISTATE + def_tristate m if GUARD + +config DEF_STRING + def_string "guarded" if GUARD + default "fallback" + +config DEF_INT + def_int 64 if GUARD + default 32 + +config DEF_HEX + def_hex 0xdead if GUARD + default 0x0 diff --git a/scripts/kconfig/tests/def_type/__init__.py b/scripts/kconfig/tests/def_type/__init__.py new file mode 100644 index 00000000000000..1ebaf5da07c821 --- /dev/null +++ b/scripts/kconfig/tests/def_type/__init__.py @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0 +""" +Set a symbol's type and its default value in one line. + +"def_bool", "def_tristate", "def_string", "def_int" and "def_hex" are +shorthand for a type definition plus a "default" property. Check that +each one sets the type, and that an "if" on the shorthand does not +disturb the usual default cascade: the shorthand is only the first arm +of the list, so a later "default" still applies when its condition is +not met. +""" + +def test(conf): + assert conf.olddefconfig(dot_config='guard_y.config') == 0 + assert conf.config_matches('expected_guard_y') + + assert conf.olddefconfig(dot_config='guard_n.config') == 0 + assert conf.config_matches('expected_guard_n') diff --git a/scripts/kconfig/tests/def_type/expected_guard_n b/scripts/kconfig/tests/def_type/expected_guard_n new file mode 100644 index 00000000000000..14719720a8b931 --- /dev/null +++ b/scripts/kconfig/tests/def_type/expected_guard_n @@ -0,0 +1,9 @@ +# +# Automatically generated file; DO NOT EDIT. +# Main menu +# +CONFIG_MODULES=y +# CONFIG_GUARD is not set +CONFIG_DEF_STRING="fallback" +CONFIG_DEF_INT=32 +CONFIG_DEF_HEX=0x0 diff --git a/scripts/kconfig/tests/def_type/expected_guard_y b/scripts/kconfig/tests/def_type/expected_guard_y new file mode 100644 index 00000000000000..b2844072d0b8f4 --- /dev/null +++ b/scripts/kconfig/tests/def_type/expected_guard_y @@ -0,0 +1,11 @@ +# +# Automatically generated file; DO NOT EDIT. +# Main menu +# +CONFIG_MODULES=y +CONFIG_GUARD=y +CONFIG_DEF_BOOL=y +CONFIG_DEF_TRISTATE=m +CONFIG_DEF_STRING="guarded" +CONFIG_DEF_INT=64 +CONFIG_DEF_HEX=0xdead diff --git a/scripts/kconfig/tests/def_type/guard_n.config b/scripts/kconfig/tests/def_type/guard_n.config new file mode 100644 index 00000000000000..ed9ad6c1a2d4ec --- /dev/null +++ b/scripts/kconfig/tests/def_type/guard_n.config @@ -0,0 +1 @@ +# CONFIG_GUARD is not set diff --git a/scripts/kconfig/tests/def_type/guard_y.config b/scripts/kconfig/tests/def_type/guard_y.config new file mode 100644 index 00000000000000..afe35b0845425f --- /dev/null +++ b/scripts/kconfig/tests/def_type/guard_y.config @@ -0,0 +1 @@ +CONFIG_GUARD=y diff --git a/scripts/kconfig/tests/err_num_bounds/Kconfig b/scripts/kconfig/tests/err_num_bounds/Kconfig new file mode 100644 index 00000000000000..c439366c03b676 --- /dev/null +++ b/scripts/kconfig/tests/err_num_bounds/Kconfig @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: GPL-2.0 +# Test bounds checks for 'int' and 'hex' constants + +config INT_SOURCE + int + +config HEX_SOURCE + hex + +config BOOL_SOURCE + bool + +# Valid values at the limits of the type + +config INT_MIN + int + default -9223372036854775808 + +config INT_MAX + int + default 9223372036854775807 + +config HEX_MIN + hex + default 0x0 + +config HEX_MAX + hex + default 0xffffffffffffffff + +config INT_RANGE_LIMITS + int + range -9223372036854775808 9223372036854775807 + +config HEX_RANGE_LIMITS + hex + range 0 0xffffffffffffffff + +config INT_FROM_INT + int + default INT_SOURCE + +config HEX_FROM_HEX + hex + default HEX_SOURCE + +# Constants outside the bounds + +config INT_DEFAULT_TOO_HIGH + int + default 10000000000000000000 + +config INT_DEFAULT_TOO_LOW + int + default -9223372036854775809 + +config INT_RANGE_TOO_HIGH + int + range 0 10000000000000000000 + +config INT_RANGE_TOO_LOW + int + range -10000000000000000000 0 + +config INT_RANGE_BOTH_OUTSIDE + int + range -9223372036854775809 10000000000000000000 + +config HEX_DEFAULT_TOO_HIGH + hex + default 0x10000000000000000 + +config HEX_RANGE_TOO_HIGH + hex + range 0 0x10000000000000000 + +config HEX_RANGE_BOTH_TOO_HIGH + hex + range 0x10000000000000000 0x20000000000000000 diff --git a/scripts/kconfig/tests/err_num_bounds/__init__.py b/scripts/kconfig/tests/err_num_bounds/__init__.py new file mode 100644 index 00000000000000..72ac6aa244918b --- /dev/null +++ b/scripts/kconfig/tests/err_num_bounds/__init__.py @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0 +""" +Detect constants outside the 'int' and 'hex' bounds. + +An int constant must fit in a signed 64-bit integer, and a hex constant must +fit in an unsigned 64-bit integer. +""" + + +def test(conf): + assert conf.olddefconfig() == 1 + assert conf.stderr_matches('expected_stderr') diff --git a/scripts/kconfig/tests/err_num_bounds/expected_stderr b/scripts/kconfig/tests/err_num_bounds/expected_stderr new file mode 100644 index 00000000000000..5f89cf4afff8ac --- /dev/null +++ b/scripts/kconfig/tests/err_num_bounds/expected_stderr @@ -0,0 +1,10 @@ +Kconfig:51: error: integer constant '10000000000000000000' is outside the 64-bit signed bounds +Kconfig:55: error: integer constant '-9223372036854775809' is outside the 64-bit signed bounds +Kconfig:59: error: integer constant '10000000000000000000' is outside the 64-bit signed bounds +Kconfig:63: error: integer constant '-10000000000000000000' is outside the 64-bit signed bounds +Kconfig:67: error: integer constant '-9223372036854775809' is outside the 64-bit signed bounds +Kconfig:67: error: integer constant '10000000000000000000' is outside the 64-bit signed bounds +Kconfig:71: error: hex constant '0x10000000000000000' is outside the 64-bit unsigned bounds +Kconfig:75: error: hex constant '0x10000000000000000' is outside the 64-bit unsigned bounds +Kconfig:79: error: hex constant '0x10000000000000000' is outside the 64-bit unsigned bounds +Kconfig:79: error: hex constant '0x20000000000000000' is outside the 64-bit unsigned bounds diff --git a/scripts/kconfig/tests/err_num_mismatch/Kconfig b/scripts/kconfig/tests/err_num_mismatch/Kconfig new file mode 100644 index 00000000000000..8406f3bb6419ff --- /dev/null +++ b/scripts/kconfig/tests/err_num_mismatch/Kconfig @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: GPL-2.0 +# Test 'int' and 'hex' symbols that reference each other + +config INT_SOURCE + int + +config HEX_SOURCE + hex + +# 'hex' reference from 'int' + +config INT_DEFAULT_HEX + int + default HEX_SOURCE + +config INT_RANGE_HEX + int + range HEX_SOURCE 1 + +# A hex symbol must not reference an int symbol + +config HEX_DEFAULT_INT + hex + default INT_SOURCE + +config HEX_RANGE_INT + hex + range INT_SOURCE 0x1 + +# Referencing the same type is valid + +config INT_FROM_INT + int + default INT_SOURCE + +config HEX_FROM_HEX + hex + range 0 HEX_SOURCE diff --git a/scripts/kconfig/tests/err_num_mismatch/__init__.py b/scripts/kconfig/tests/err_num_mismatch/__init__.py new file mode 100644 index 00000000000000..275f2a6e9a5bf2 --- /dev/null +++ b/scripts/kconfig/tests/err_num_mismatch/__init__.py @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +""" +Reject direct references ('default' or 'range') between int and hex options. +""" + + +def test(conf): + assert conf.olddefconfig() == 1 + assert conf.stderr_matches('expected_stderr') diff --git a/scripts/kconfig/tests/err_num_mismatch/expected_stderr b/scripts/kconfig/tests/err_num_mismatch/expected_stderr new file mode 100644 index 00000000000000..587c34467ae632 --- /dev/null +++ b/scripts/kconfig/tests/err_num_mismatch/expected_stderr @@ -0,0 +1,4 @@ +Kconfig:14: error: 'HEX_SOURCE' is an invalid value for 'integer' +Kconfig:18: error: 'HEX_SOURCE' is an invalid value for 'integer' +Kconfig:24: error: 'INT_SOURCE' is an invalid value for 'hex' +Kconfig:28: error: 'INT_SOURCE' is an invalid value for 'hex' diff --git a/scripts/kconfig/tests/err_num_non_numeric_ref/Kconfig b/scripts/kconfig/tests/err_num_non_numeric_ref/Kconfig new file mode 100644 index 00000000000000..0ca7a4a460f6cf --- /dev/null +++ b/scripts/kconfig/tests/err_num_non_numeric_ref/Kconfig @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: GPL-2.0 +# Test non-numeric symbol references from numeric symbols + +config BOOL_SOURCE + bool + +config TRISTATE_SOURCE + tristate + +config STRING_SOURCE + string + +# Invalid int defaults + +config INT_DEFAULT_BOOL + int + default BOOL_SOURCE + +config INT_DEFAULT_TRISTATE + int + default TRISTATE_SOURCE + +config INT_DEFAULT_STRING + int + default STRING_SOURCE + +# Invalid hex defaults + +config HEX_DEFAULT_BOOL + hex + default BOOL_SOURCE + +config HEX_DEFAULT_TRISTATE + hex + default TRISTATE_SOURCE + +config HEX_DEFAULT_STRING + hex + default STRING_SOURCE + +# Invalid int ranges + +config INT_RANGE_BOOL + int + range BOOL_SOURCE 1 + +config INT_RANGE_TRISTATE + int + range TRISTATE_SOURCE 1 + +config INT_RANGE_STRING + int + range STRING_SOURCE 1 + +config INT_RANGE_MULTIPLE + int + range BOOL_SOURCE TRISTATE_SOURCE + +# Invalid hex ranges + +config HEX_RANGE_BOOL + hex + range BOOL_SOURCE 0x1 + +config HEX_RANGE_TRISTATE + hex + range TRISTATE_SOURCE 0x1 + +config HEX_RANGE_STRING + hex + range STRING_SOURCE 0x1 + +config HEX_RANGE_MULTIPLE + hex + range BOOL_SOURCE TRISTATE_SOURCE diff --git a/scripts/kconfig/tests/err_num_non_numeric_ref/__init__.py b/scripts/kconfig/tests/err_num_non_numeric_ref/__init__.py new file mode 100644 index 00000000000000..9632907abead8b --- /dev/null +++ b/scripts/kconfig/tests/err_num_non_numeric_ref/__init__.py @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +""" +Reject nonnumeric symbol references from int and hex properties. +""" + + +def test(conf): + assert conf.olddefconfig() == 1 + assert conf.stderr_matches('expected_stderr') diff --git a/scripts/kconfig/tests/err_num_non_numeric_ref/expected_stderr b/scripts/kconfig/tests/err_num_non_numeric_ref/expected_stderr new file mode 100644 index 00000000000000..005f855ecbddf4 --- /dev/null +++ b/scripts/kconfig/tests/err_num_non_numeric_ref/expected_stderr @@ -0,0 +1,16 @@ +Kconfig:17: error: 'BOOL_SOURCE' is an invalid value for 'integer' +Kconfig:21: error: 'TRISTATE_SOURCE' is an invalid value for 'integer' +Kconfig:25: error: 'STRING_SOURCE' is an invalid value for 'integer' +Kconfig:31: error: 'BOOL_SOURCE' is an invalid value for 'hex' +Kconfig:35: error: 'TRISTATE_SOURCE' is an invalid value for 'hex' +Kconfig:39: error: 'STRING_SOURCE' is an invalid value for 'hex' +Kconfig:45: error: 'BOOL_SOURCE' is an invalid value for 'integer' +Kconfig:49: error: 'TRISTATE_SOURCE' is an invalid value for 'integer' +Kconfig:53: error: 'STRING_SOURCE' is an invalid value for 'integer' +Kconfig:57: error: 'BOOL_SOURCE' is an invalid value for 'integer' +Kconfig:57: error: 'TRISTATE_SOURCE' is an invalid value for 'integer' +Kconfig:63: error: 'BOOL_SOURCE' is an invalid value for 'hex' +Kconfig:67: error: 'TRISTATE_SOURCE' is an invalid value for 'hex' +Kconfig:71: error: 'STRING_SOURCE' is an invalid value for 'hex' +Kconfig:75: error: 'BOOL_SOURCE' is an invalid value for 'hex' +Kconfig:75: error: 'TRISTATE_SOURCE' is an invalid value for 'hex' diff --git a/scripts/kconfig/tests/err_recursive_dep/Kconfig b/scripts/kconfig/tests/err_recursive_dep/Kconfig index ebdb3ffd87170a..bd1ff92bcab247 100644 --- a/scripts/kconfig/tests/err_recursive_dep/Kconfig +++ b/scripts/kconfig/tests/err_recursive_dep/Kconfig @@ -61,3 +61,27 @@ config G bool "G" endmenu + +# depends on and range + +config H1 + int + depends on H2 < H3 + +config H2 + int + range H3 H1 + +config H3 + int + range H1 H2 + +# range + +config I1 + int + range I2 I2 + +config I2 + int + range I1 I1 diff --git a/scripts/kconfig/tests/err_recursive_dep/expected_stderr b/scripts/kconfig/tests/err_recursive_dep/expected_stderr index fc2e860af082cb..08f8430876782b 100644 --- a/scripts/kconfig/tests/err_recursive_dep/expected_stderr +++ b/scripts/kconfig/tests/err_recursive_dep/expected_stderr @@ -36,3 +36,16 @@ error: recursive dependency detected! symbol G depends on G For a resolution refer to Documentation/kbuild/kconfig-language.rst subsection "Kconfig recursive dependency limitations" + +error: recursive dependency detected! + symbol H1 depends on H2 + symbol H2 range value contains H3 + symbol H3 range value contains H1 +For a resolution refer to Documentation/kbuild/kconfig-language.rst +subsection "Kconfig recursive dependency limitations" + +error: recursive dependency detected! + symbol I1 range value contains I2 + symbol I2 range value contains I1 +For a resolution refer to Documentation/kbuild/kconfig-language.rst +subsection "Kconfig recursive dependency limitations" diff --git a/scripts/kconfig/tests/randconfig_probability/Kconfig b/scripts/kconfig/tests/randconfig_probability/Kconfig new file mode 100644 index 00000000000000..84f4e5fcc51e4b --- /dev/null +++ b/scripts/kconfig/tests/randconfig_probability/Kconfig @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only + +config MODULES + bool + default y + modules + +config BOOL + bool "Bool" + +config TRI + tristate "Tristate" diff --git a/scripts/kconfig/tests/randconfig_probability/__init__.py b/scripts/kconfig/tests/randconfig_probability/__init__.py new file mode 100644 index 00000000000000..e6e5dda10ad929 --- /dev/null +++ b/scripts/kconfig/tests/randconfig_probability/__init__.py @@ -0,0 +1,130 @@ +# SPDX-License-Identifier: GPL-2.0-only +"""Validate KCONFIG_PROBABILITY without changing the supported distributions.""" + + +def test_malformed_warns(conf): + probabilities = [ + 'invalid', ' ', '50%', '50 ', '0x32', '10 20', '10:20x', + '10:20:invalid', '10:20:30x', + ':50', '50:', '10::20', '10:20:', '10:20:30:', '10:20:30:40', + '-0', '+0', '+', '-', '+100:0', ' \t100:0', '0: \t100:0', + ] + for probability in probabilities: + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': probability, + 'KCONFIG_SEED': '0', + }) == 0, repr(probability) + assert ('warning: KCONFIG_PROBABILITY has malformed format' in + conf.stderr), repr(probability) + assert conf.stderr.count('warning:') == 1, repr(probability) + assert conf.config is not None, repr(probability) + + +def test_malformed_preserves_config(conf): + probabilities = [('50%', '50:0:0'), ('-0', '0')] + for seed in range(20): + for probability, equivalent in probabilities: + context = 'probability={!r}, equivalent={!r}, seed={}'.format( + probability, equivalent, seed) + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': probability, + 'KCONFIG_SEED': hex(seed), + }) == 0, context + assert ('warning: KCONFIG_PROBABILITY has malformed format' in + conf.stderr), context + assert conf.stderr.count('warning:') == 1, context + malformed = conf.config + + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': equivalent, + 'KCONFIG_SEED': hex(seed), + }) == 0, context + assert 'warning:' not in conf.stderr, context + assert conf.config == malformed, context + + +def test_werror(conf): + probabilities = [ + ('', 0), ('50', 0), ('50%', 1), ('-0', 1), ('10:20:30:40', 1), + ] + for probability, status in probabilities: + for werror in ['', '0', '1']: + context = 'probability={!r}, werror={!r}'.format( + probability, werror) + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': probability, + 'KCONFIG_SEED': '0', + 'KCONFIG_WERROR': werror, + }) == status, context + if status: + assert ('warning: KCONFIG_PROBABILITY has malformed format' in + conf.stderr), context + assert conf.stderr.count('warning:') == 1, context + else: + assert 'warning:' not in conf.stderr, context + + +def test_out_of_range(conf): + probabilities = [ + '-1', '101', '+101', '0:101', '0:0:101', '60:41', '0:60:41', + '4294967296', '-4294967296', + '999999999999999999999999', '-999999999999999999999999', + ] + for probability in probabilities: + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': probability, + 'KCONFIG_SEED': '0', + }) == 1, repr(probability) + assert 'KCONFIG_PROBABILITY:' in conf.stderr, repr(probability) + + +def test_valid(conf): + probabilities = [ + ('0', 'n', 'n'), + ('0:0', 'n', 'n'), + ('100:0', 'y', 'y'), + ('0:100', 'y', 'm'), + ('100:0:0', 'y', 'n'), + ('0:100:0', 'n', 'y'), + ('0:0:100', 'n', 'm'), + ('000:000:100', 'n', 'm'), + ] + for probability, boolean, tristate in probabilities: + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': probability, + 'KCONFIG_SEED': '0', + }) == 0, repr(probability) + assert 'warning:' not in conf.stderr, repr(probability) + for symbol, value in [('BOOL', boolean), ('TRI', tristate)]: + if value == 'n': + expected = '# CONFIG_{} is not set'.format(symbol) + else: + expected = 'CONFIG_{}={}'.format(symbol, value) + assert expected in conf.config.splitlines(), repr(probability) + + +def test_single_probability_matches_tristate_split(conf): + for seed in range(20): + context = 'seed={}'.format(seed) + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': '50', + 'KCONFIG_SEED': hex(seed), + }) == 0, context + assert 'warning:' not in conf.stderr, context + single = conf.config + + # 50% boolean y; 25% tristate y, 25% m, and 50% n. + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': '50:25:25', + 'KCONFIG_SEED': hex(seed), + }) == 0, context + assert 'warning:' not in conf.stderr, context + assert conf.config == single, context + + +def test_empty(conf): + assert conf._run_conf('--randconfig', extra_env={ + 'KCONFIG_PROBABILITY': '', + 'KCONFIG_SEED': '0', + }) == 0 + assert 'warning:' not in conf.stderr diff --git a/scripts/kconfig/tests/savedefconfig_range/Kconfig b/scripts/kconfig/tests/savedefconfig_range/Kconfig new file mode 100644 index 00000000000000..d879870a526750 --- /dev/null +++ b/scripts/kconfig/tests/savedefconfig_range/Kconfig @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: GPL-2.0 + +# Static default and range values + +config INT_DEFAULT_ABOVE_RANGE + int + range 1 1 + default 16 + +config INT_DEFAULT_BELOW_RANGE + int + range 4 8 + default 2 + +# Default and range values determined by other options + +config RANGE_UPPER_BOUND + int + default 3 + +config INT_DYNAMIC_RANGE + int + range 0 RANGE_UPPER_BOUND + default 4 + +# Hex + +config HEX_DEFAULT_ABOVE_RANGE + hex + range 0x10 0x20 + default 0x40 + +# Implicit default value of 0 + +config INT_IMPLICIT_DEFAULT_ZERO + int + range 1 4 + +# Conditional range + +config USE_FIRST_RANGE + bool + default y + +config INT_CONDITIONAL_RANGE + int + range 1 2 if USE_FIRST_RANGE + range 3 4 if !USE_FIRST_RANGE + default 3 + +# Conditional default + +config USE_DEFAULT + bool + default y + +config INT_CONDITIONAL_DEFAULT + int + range 1 2 + default 3 if USE_DEFAULT + +# Default outside of the range of a symbol with a prompt + +config INT_PROMPT_DEFAULT_ABOVE_RANGE + int "int with prompt" + range 1 4 + default 8 diff --git a/scripts/kconfig/tests/savedefconfig_range/__init__.py b/scripts/kconfig/tests/savedefconfig_range/__init__.py new file mode 100644 index 00000000000000..961454be732c2f --- /dev/null +++ b/scripts/kconfig/tests/savedefconfig_range/__init__.py @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 +"""Test savedefconfig with numerical defaults outside of ranges.""" + + +def test(conf): + assert conf._run_conf('--savedefconfig=defconfig', dot_config='config', + out_file='defconfig') == 0 + assert conf.config_matches('expected_defconfig') diff --git a/scripts/kconfig/tests/savedefconfig_range/config b/scripts/kconfig/tests/savedefconfig_range/config new file mode 100644 index 00000000000000..079a3107a02b62 --- /dev/null +++ b/scripts/kconfig/tests/savedefconfig_range/config @@ -0,0 +1,8 @@ +CONFIG_INT_DEFAULT_ABOVE_RANGE=1 +CONFIG_INT_DEFAULT_BELOW_RANGE=4 +CONFIG_INT_DYNAMIC_RANGE=3 +CONFIG_HEX_DEFAULT_ABOVE_RANGE=0x20 +CONFIG_INT_IMPLICIT_DEFAULT_ZERO=1 +CONFIG_INT_CONDITIONAL_RANGE=2 +CONFIG_INT_CONDITIONAL_DEFAULT=2 +CONFIG_INT_PROMPT_DEFAULT_ABOVE_RANGE=4 diff --git a/scripts/kconfig/tests/savedefconfig_range/expected_defconfig b/scripts/kconfig/tests/savedefconfig_range/expected_defconfig new file mode 100644 index 00000000000000..e69de29bb2d1d6 diff --git a/scripts/kconfig/tests/warn_changed_input/Kconfig b/scripts/kconfig/tests/warn_changed_input/Kconfig index 69845e2f3fb34b..df4183a08420f9 100644 --- a/scripts/kconfig/tests/warn_changed_input/Kconfig +++ b/scripts/kconfig/tests/warn_changed_input/Kconfig @@ -26,6 +26,11 @@ config NUM Kconfig resolves it to the constrained in-range value. The warning should report that adjustment. +config HEX_RANGE + hex + range 0 0xfffffffffffffffe + default 0xffffffffffffffff + config DUP bool "DUP" depends on DEP diff --git a/scripts/kconfig/tests/warn_changed_input/__init__.py b/scripts/kconfig/tests/warn_changed_input/__init__.py index 4c3bca6af846e0..17b602dd280e88 100644 --- a/scripts/kconfig/tests/warn_changed_input/__init__.py +++ b/scripts/kconfig/tests/warn_changed_input/__init__.py @@ -8,26 +8,21 @@ def test(conf): + warn_changed_input = { + "KCONFIG_WARN_CHANGED_INPUT": "1", + } + assert conf.olddefconfig('config') == 0 assert 'user-provided values changed by Kconfig' not in conf.stderr - assert conf._run_conf('--olddefconfig', dot_config='config', - extra_env={ - 'KCONFIG_WARN_CHANGED_INPUT': '1', - }) == 0 + assert conf.olddefconfig('config', extra_env=warn_changed_input) == 0 assert conf.stderr_contains('expected_stderr') assert conf.config_matches('expected_config') - assert conf._run_conf('--olddefconfig', dot_config='config', - extra_env={ - 'KCONFIG_WARN_CHANGED_INPUT': '1', - }, silent=True) == 0 + assert conf.olddefconfig('config', extra_env=warn_changed_input, + silent=True) == 0 assert conf.stderr_contains('expected_stderr') - assert conf._run_conf('--savedefconfig=defconfig', dot_config='config', - out_file='defconfig', - extra_env={ - 'KCONFIG_WARN_CHANGED_INPUT': '1', - }) == 0 + assert conf.savedefconfig('config', extra_env=warn_changed_input) == 0 assert conf.stderr_contains('expected_stderr') assert conf.config_matches('expected_defconfig') diff --git a/scripts/kconfig/tests/warn_changed_input/expected_config b/scripts/kconfig/tests/warn_changed_input/expected_config index fe8bbec66c5358..6a8d98a69b7e77 100644 --- a/scripts/kconfig/tests/warn_changed_input/expected_config +++ b/scripts/kconfig/tests/warn_changed_input/expected_config @@ -4,3 +4,4 @@ # # CONFIG_DEP is not set CONFIG_NUM=20 +CONFIG_HEX_RANGE=0xfffffffffffffffe diff --git a/scripts/kconfig/tests/warn_num_bounds/Kconfig b/scripts/kconfig/tests/warn_num_bounds/Kconfig new file mode 100644 index 00000000000000..a810225a58a397 --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/Kconfig @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: GPL-2.0 + +mainmenu "Numeric bounds test" + +config INT_TOO_LOW + int "Integer below its type bounds" + +config INT_TOO_HIGH + int "Integer above its type bounds" + +config HEX_TOO_HIGH + hex "Hex value above its type bounds" + +config INT_MIN + int "Minimum valid integer" + +config INT_MAX + int "Maximum valid integer" + +config HEX_MIN + hex "Minimum valid hex value" + +config HEX_MAX + hex "Maximum valid hex value" diff --git a/scripts/kconfig/tests/warn_num_bounds/__init__.py b/scripts/kconfig/tests/warn_num_bounds/__init__.py new file mode 100644 index 00000000000000..db75182582178b --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/__init__.py @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: GPL-2.0 +"""Test user values outside the numeric type bounds.""" + + +def test(conf): + in_keys = ( + '-9223372036854775809\n' + '-1\n' + '9223372036854775808\n' + '1\n' + '0x10000000000000000\n' + '0x1\n' + '-9223372036854775808\n' + '9223372036854775807\n' + '0x0\n' + '0xffffffffffffffff\n' + ) + + assert conf.oldaskconfig(in_keys=in_keys) == 0 + assert conf.stderr_matches('expected_frontend_stderr') + assert conf.config_matches('expected_frontend_config') + + assert conf.olddefconfig('config') == 0 + assert conf.stderr_matches('expected_config_stderr') + assert conf.config_matches('expected_config') diff --git a/scripts/kconfig/tests/warn_num_bounds/config b/scripts/kconfig/tests/warn_num_bounds/config new file mode 100644 index 00000000000000..74bf263f99bf23 --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/config @@ -0,0 +1,7 @@ +CONFIG_INT_TOO_LOW=-9223372036854775809 +CONFIG_INT_TOO_HIGH=9223372036854775808 +CONFIG_HEX_TOO_HIGH=0x10000000000000000 +CONFIG_INT_MIN=-9223372036854775808 +CONFIG_INT_MAX=9223372036854775807 +CONFIG_HEX_MIN=0x0 +CONFIG_HEX_MAX=0xffffffffffffffff diff --git a/scripts/kconfig/tests/warn_num_bounds/expected_config b/scripts/kconfig/tests/warn_num_bounds/expected_config new file mode 100644 index 00000000000000..8b1aef612bc6b2 --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/expected_config @@ -0,0 +1,11 @@ +# +# Automatically generated file; DO NOT EDIT. +# Numeric bounds test +# +CONFIG_INT_TOO_LOW=-9223372036854775809 +CONFIG_INT_TOO_HIGH=9223372036854775808 +CONFIG_HEX_TOO_HIGH=0x10000000000000000 +CONFIG_INT_MIN=-9223372036854775808 +CONFIG_INT_MAX=9223372036854775807 +CONFIG_HEX_MIN=0x0 +CONFIG_HEX_MAX=0xffffffffffffffff diff --git a/scripts/kconfig/tests/warn_num_bounds/expected_config_stderr b/scripts/kconfig/tests/warn_num_bounds/expected_config_stderr new file mode 100644 index 00000000000000..be2ed7fbf64851 --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/expected_config_stderr @@ -0,0 +1,3 @@ +.config:1:warning: value '-9223372036854775809' for INT_TOO_LOW is outside the 64-bit signed integer bounds +.config:2:warning: value '9223372036854775808' for INT_TOO_HIGH is outside the 64-bit signed integer bounds +.config:3:warning: value '0x10000000000000000' for HEX_TOO_HIGH is outside the 64-bit unsigned integer bounds diff --git a/scripts/kconfig/tests/warn_num_bounds/expected_frontend_config b/scripts/kconfig/tests/warn_num_bounds/expected_frontend_config new file mode 100644 index 00000000000000..a7e842517026ef --- /dev/null +++ b/scripts/kconfig/tests/warn_num_bounds/expected_frontend_config @@ -0,0 +1,11 @@ +# +# Automatically generated file; DO NOT EDIT. +# Numeric bounds test +# +CONFIG_INT_TOO_LOW=-1 +CONFIG_INT_TOO_HIGH=1 +CONFIG_HEX_TOO_HIGH=0x1 +CONFIG_INT_MIN=-9223372036854775808 +CONFIG_INT_MAX=9223372036854775807 +CONFIG_HEX_MIN=0x0 +CONFIG_HEX_MAX=0xffffffffffffffff diff --git a/scripts/kconfig/tests/warn_num_bounds/expected_frontend_stderr b/scripts/kconfig/tests/warn_num_bounds/expected_frontend_stderr new file mode 100644 index 00000000000000..e69de29bb2d1d6 diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh index ab0b8125c8cbc6..c520741f85c2d4 100755 --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -96,6 +96,11 @@ vmlinux_link() ldflags="${ldflags} ${wl}--strip-debug" fi + # Only the final link actually requires the relocations. + if [ "${output}" = "${VMLINUX}" ] && is_enabled CONFIG_ARCH_VMLINUX_NEEDS_RELOCS; then + ldflags="${ldflags} ${wl}--emit-relocs" + fi + if [ -n "${generate_map}" ]; then ldflags="${ldflags} ${wl}-Map=vmlinux.map" fi @@ -118,11 +123,15 @@ kallsymso_changed() ! cmp -s "${kallsymso_prev}.sym" "${kallsymso}.sym" } -# Create ${2}.o file with all symbols from the ${1} object file +# Create ${2}.o with the kallsyms tables for ${1} (the vmlinux, or an empty +# listing for the first pass); list the symbols used in ${3} if given. kallsyms() { local kallsymopt; + if [ -n "${3:-}" ]; then + kallsymopt="--sysmap=${3}" + fi if is_enabled CONFIG_KALLSYMS_ALL; then kallsymopt="${kallsymopt} --all-symbols" fi @@ -132,7 +141,7 @@ kallsyms() fi info KSYMS "${2}.S" - scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S" + scripts/kallsyms ${kallsymopt} "${1}" "${2}.bin" > "${2}.S" info AS "${2}.o" ${CC} ${NOSTDINC_FLAGS} ${LINUXINCLUDE} ${KBUILD_CPPFLAGS} \ @@ -145,23 +154,20 @@ kallsyms() # Perform kallsyms for the given temporary vmlinux. sysmap_and_kallsyms() { - mksysmap "${1}" "${1}.syms" - kallsyms "${1}.syms" "${1}.kallsyms" - + kallsyms "${1}" "${1}.kallsyms" "${1}.syms" kallsyms_sysmap=${1}.syms } # Create map file with all symbols from ${1} -# See mksymap for additional details mksysmap() { - info NM ${2} - ${NM} -n "${1}" | sed -f "${srctree}/scripts/mksysmap" > "${2}" + info SYSMAP ${2} + scripts/kallsyms --sysmap="${2}" "${1}" } sorttable() { - ${NM} -S ${1} > .tmp_vmlinux.nm-sort + ${NM} -p -S ${1} > .tmp_vmlinux.nm-sort ${objtree}/scripts/sorttable -s .tmp_vmlinux.nm-sort ${1} } diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 15ba26974fd729..9bd35aa721191f 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -292,7 +292,8 @@ def process_dtb(fname, args): if 'scripts/dtc/fdtoverlay' in cmd: # This depends on the structure of the composite DTB command - files = cmd.split() + # Ignore any subsequent commands, such as schema validation + files = cmd.split(';', 1)[0].split() files = files[files.index('-i') + 1:] else: files = [fname] diff --git a/scripts/mkcompile_h b/scripts/mkcompile_h index 2596f78e52ef95..bf0d5669d9bcd5 100755 --- a/scripts/mkcompile_h +++ b/scripts/mkcompile_h @@ -11,7 +11,7 @@ else LINUX_COMPILE_BY=$KBUILD_BUILD_USER fi if test -z "$KBUILD_BUILD_HOST"; then - LINUX_COMPILE_HOST=`uname -n` + LINUX_COMPILE_HOST=$(uname -n) else LINUX_COMPILE_HOST=$KBUILD_BUILD_HOST fi diff --git a/scripts/mksysmap b/scripts/mksysmap deleted file mode 100755 index 856b26ba2ac024..00000000000000 --- a/scripts/mksysmap +++ /dev/null @@ -1,94 +0,0 @@ -#!/bin/sed -f -# SPDX-License-Identifier: GPL-2.0-only -# -# sed script to filter out symbols that are not needed for System.map, -# or not suitable for kallsyms. The input should be 'nm -n '. -# -# System.map is used by module-init tools and some debugging -# tools to retrieve the actual addresses of symbols in the kernel. -# -# readprofile starts reading symbols when _stext is found, and -# continue until it finds a symbol which is not either of 'T', 't', -# 'W' or 'w'. -# -# --------------------------------------------------------------------------- -# Ignored symbol types -# - -# a: local absolute symbols -# N: debugging symbols -# U: undefined global symbols -# w: local weak symbols -/ [aNUw] /d - -# --------------------------------------------------------------------------- -# Ignored prefixes -# (do not forget a space before each pattern) - -# local symbols for ARM, MIPS, etc. -/ \$/d - -# local labels, .LBB, .Ltmpxxx, .L__unnamed_xx, .LASANPC, etc. -/ \.L/d - -# arm64 EFI stub namespace -/ __efistub_/d - -# arm64 local symbols in PIE namespace -/ __pi_\$/d -/ __pi_\.L/d - -# arm64 local symbols in non-VHE KVM namespace -/ __kvm_nvhe_\$/d -/ __kvm_nvhe_\.L/d - -# lld arm/aarch64/mips thunks -/ __[[:alnum:]]*Thunk_/d - -# CFI type identifiers -/ __kcfi_typeid_/d -/ __kvm_nvhe___kcfi_typeid_/d -/ __pi___kcfi_typeid_/d - -# CRC from modversions -/ __crc_/d - -# EXPORT_SYMBOL (symbol name) -/ __kstrtab_/d - -# EXPORT_SYMBOL (namespace) -/ __kstrtabns_/d - -# MODULE_DEVICE_TABLE (symbol name) -/ __mod_device_table__/d - -# --------------------------------------------------------------------------- -# Ignored suffixes -# (do not forget '$' after each pattern) - -# arm -/_from_arm$/d -/_from_thumb$/d -/_veneer$/d - -# --------------------------------------------------------------------------- -# Ignored symbols (exact match) -# (do not forget a space before and '$' after each pattern) - -# for LoongArch? -/ L0$/d - -# ppc -/ _SDA_BASE_$/d -/ _SDA2_BASE_$/d - -# MODULE_INFO() -/ __UNIQUE_ID_modinfo_[0-9]*$/d - -# --------------------------------------------------------------------------- -# Ignored patterns -# (symbols that contain the pattern are ignored) - -# ppc stub -/\.long_branch\./d -/\.plt_branch\./d diff --git a/scripts/mod/devicetable-offsets.c b/scripts/mod/devicetable-offsets.c index b4178c42d08f50..1492e9fdc7cb46 100644 --- a/scripts/mod/devicetable-offsets.c +++ b/scripts/mod/devicetable-offsets.c @@ -254,6 +254,9 @@ int main(void) DEVID(auxiliary_device_id); DEVID_FIELD(auxiliary_device_id, name); + DEVID(arm_smccc_device_id); + DEVID_FIELD(arm_smccc_device_id, func_id); + DEVID(ssam_device_id); DEVID_FIELD(ssam_device_id, match_flags); DEVID_FIELD(ssam_device_id, domain); diff --git a/scripts/mod/file2alias.c b/scripts/mod/file2alias.c index 8d36c74dec2d55..980d7da456bbab 100644 --- a/scripts/mod/file2alias.c +++ b/scripts/mod/file2alias.c @@ -1349,6 +1349,13 @@ static void do_auxiliary_entry(struct module *mod, void *symval) module_alias_printf(mod, false, AUXILIARY_MODULE_PREFIX "%s", *name); } +static void do_arm_smccc_entry(struct module *mod, void *symval) +{ + DEF_FIELD(symval, arm_smccc_device_id, func_id); + + module_alias_printf(mod, false, ARM_SMCCC_MODULE_PREFIX "f%08X", func_id); +} + /* * Looks like: ssam:dNcNtNiNfN * @@ -1519,6 +1526,7 @@ static const struct devtable devtable[] = { {"mhi", SIZE_mhi_device_id, do_mhi_entry}, {"mhi_ep", SIZE_mhi_device_id, do_mhi_ep_entry}, {"auxiliary", SIZE_auxiliary_device_id, do_auxiliary_entry}, + {"arm_smccc", SIZE_arm_smccc_device_id, do_arm_smccc_entry}, {"ssam", SIZE_ssam_device_id, do_ssam_entry}, {"dfl", SIZE_dfl_device_id, do_dfl_entry}, {"ishtp", SIZE_ishtp_device_id, do_ishtp_entry}, diff --git a/scripts/remove-stale-files b/scripts/remove-stale-files index 6e39fa8540df2f..70a2ead4d0f128 100755 --- a/scripts/remove-stale-files +++ b/scripts/remove-stale-files @@ -26,3 +26,7 @@ rm -f scripts/selinux/genheaders/genheaders rm -f *.spec rm -f lib/test_fortify.log + +rm -f rust/doctests_kernel_generated_kunit.c + +rm -f usr/gen_init_cpio diff --git a/scripts/rustdoc_test_builder.rs b/scripts/rustdoc_test_builder.rs index df864437cef70c..0361e22853f212 100644 --- a/scripts/rustdoc_test_builder.rs +++ b/scripts/rustdoc_test_builder.rs @@ -72,6 +72,13 @@ fn main() { // Figure out a smaller test name based on the generated function name. let name = rustdoc_function_name.split_once("_rust_kernel_").unwrap().1; + // The `rustdoc` function name can include the absolute path when building with `O=` which is + // undesirable and creates overlong symbol names. Remap it to use a relative path. + // TODO: Remove when bumping the minimum version since with Rust >= 1.95 we already do the + // path remapping via the `rustdoc` `--remap-path-prefix` flag. + let trimmed_function_name = format!("_doctest_main_rust_kernel_{name}"); + let body = body.replace(&rustdoc_function_name, &trimmed_function_name); + let path = format!("rust/test/doctests/kernel/{name}"); std::fs::write(path, body.as_bytes()).unwrap(); diff --git a/scripts/rustdoc_test_gen.rs b/scripts/rustdoc_test_gen.rs index d087c0d9fcb3bd..754c5aafcce441 100644 --- a/scripts/rustdoc_test_gen.rs +++ b/scripts/rustdoc_test_gen.rs @@ -125,9 +125,7 @@ fn main() { // Sort paths. paths.sort(); - let mut rust_tests = String::new(); - let mut c_test_declarations = String::new(); - let mut c_test_cases = String::new(); + let mut tests = String::new(); let mut body = String::new(); let mut last_file = String::new(); let mut number = 0; @@ -172,28 +170,22 @@ fn main() { use std::fmt::Write; write!( - rust_tests, + tests, r#"/// Generated `{name}` KUnit test case from a Rust documentation test. -#[no_mangle] -pub extern "C" fn {kunit_name}(__kunit_test: *mut ::kernel::bindings::kunit) {{ - /// Overrides the usual [`assert!`] macro with one that calls KUnit instead. +#[test] +fn {kunit_name}() {{ + /// Overrides the usual [`file!`] macro with one that expands to the real path. #[allow(unused)] - macro_rules! assert {{ - ($cond:expr $(,)?) => {{{{ - ::kernel::kunit_assert!( - "{kunit_name}", c"{real_path}", __DOCTEST_ANCHOR - {line}, $cond - ); - }}}} + macro_rules! file {{ + () => {{ "{real_path}" }} }} - /// Overrides the usual [`assert_eq!`] macro with one that calls KUnit instead. + /// Overrides the usual [`line!`] macro with one that expands to the real line number. #[allow(unused)] - macro_rules! assert_eq {{ - ($left:expr, $right:expr $(,)?) => {{{{ - ::kernel::kunit_assert_eq!( - "{kunit_name}", c"{real_path}", __DOCTEST_ANCHOR - {line}, $left, $right - ); - }}}} + macro_rules! line {{ + // NOTE: This does not expand to a literal, but a constant expression. + // Therefore code that depends on `line!()` expanding to a literal needs special adjustment. + () => {{ const {{ ::core::line!() - __DOCTEST_ANCHOR + {line} }} }} }} // Many tests need the prelude, so provide it by default. @@ -213,7 +205,7 @@ pub extern "C" fn {kunit_name}(__kunit_test: *mut ::kernel::bindings::kunit) {{ /// The anchor where the test code body starts. #[allow(unused)] - static __DOCTEST_ANCHOR: i32 = ::core::line!() as i32 + {body_offset} + 2; + static __DOCTEST_ANCHOR: u32 = ::core::line!() + {body_offset} + 2; {{ #![allow(unreachable_pub, clippy::disallowed_names)] {body} @@ -224,14 +216,9 @@ pub extern "C" fn {kunit_name}(__kunit_test: *mut ::kernel::bindings::kunit) {{ "# ) .unwrap(); - - write!(c_test_declarations, "void {kunit_name}(struct kunit *);\n").unwrap(); - write!(c_test_cases, " KUNIT_CASE({kunit_name}),\n").unwrap(); } - let rust_tests = rust_tests.trim(); - let c_test_declarations = c_test_declarations.trim(); - let c_test_cases = c_test_cases.trim(); + let tests = tests.trim(); write!( BufWriter::new(File::create("rust/doctests_kernel_generated.rs").unwrap()), @@ -257,34 +244,10 @@ impl ModuleMetadata for LocalModule {{ }}; }} -{rust_tests} -"# - ) - .unwrap(); - - write!( - BufWriter::new(File::create("rust/doctests_kernel_generated_kunit.c").unwrap()), - r#"/* - * `kernel` crate documentation tests. - */ - -#include - -{c_test_declarations} - -static struct kunit_case test_cases[] = {{ - {c_test_cases} - {{ }} -}}; - -static struct kunit_suite test_suite = {{ - .name = "rust_doctests_kernel", - .test_cases = test_cases, -}}; - -kunit_test_suite(test_suite); - -MODULE_LICENSE("GPL"); +#[kernel::macros::kunit_tests(rust_doctests_kernel)] +mod tests {{ +{tests} +}} "# ) .unwrap(); diff --git a/scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py b/scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py index d7776f072e0342..93787a1cd1566d 100644 --- a/scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py +++ b/scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py @@ -338,7 +338,7 @@ def test_xen_hypercalls(self): # gen_initramfs.sh command tests def test_gen_initramfs(self): - cmd = "sh ../usr/gen_initramfs.sh -o usr/initramfs_data.cpio -l usr/.initramfs_data.cpio.d ../usr/default_cpio_list" + cmd = "sh ../scripts/gen_initramfs.sh -o usr/initramfs_data.cpio -l usr/.initramfs_data.cpio.d ../usr/default_cpio_list" expected = "../usr/default_cpio_list" self._assert_parsing(cmd, expected) diff --git a/scripts/setlocalversion b/scripts/setlocalversion index 28169d7e143b69..f264c2460ad779 100755 --- a/scripts/setlocalversion +++ b/scripts/setlocalversion @@ -33,7 +33,7 @@ if test $# -gt 0 -o ! -d "$srctree"; then fi try_tag() { - tag="$1" + local tag="$1" # Is $tag an annotated tag? if [ "$(git cat-file -t "$tag" 2> /dev/null)" != tag ]; then @@ -59,7 +59,6 @@ scm_version() { local short=false local no_dirty=false - local tag while [ $# -gt 0 ]; do diff --git a/scripts/sorttable.c b/scripts/sorttable.c index d7b50581c7329f..88e49a5c425127 100644 --- a/scripts/sorttable.c +++ b/scripts/sorttable.c @@ -322,10 +322,23 @@ static int parse_symbols(const char *fname) return -1; } - while (fscanf(fp, "%16s %16s %c %*s\n", addr_str, size_str, &type) == 3) { + while (fscanf(fp, "%16s %16s %c%*[^\n]", addr_str, size_str, &type) == 3) { uint64_t addr; uint64_t size; + /* + * nm outputs size-less entries with a missing 2nd value, which + * means fscanf() just misread its fields. + * + * These don't matter as a call site cannot be in an empty + * function, so just skip them. + * + * nm pads address and size to the same width, so if their + * widths differ, this is a misread. + */ + if (strlen(size_str) != strlen(addr_str)) + continue; + /* Only care about functions */ if (type != 't' && type != 'T' && type != 'W') continue; diff --git a/scripts/ver_linux b/scripts/ver_linux index 00bdaf30d590d5..e06d7c650710c7 100755 --- a/scripts/ver_linux +++ b/scripts/ver_linux @@ -57,6 +57,7 @@ BEGIN { printversion("pahole", version("pahole --version")) printversion("Pcmcia-cs", version("cardmgr -V")) printversion("pcmciautils", version("pccardctl -V")) + printversion("pigz", version("pigz --version")) printversion("PPP", version("pppd --version")) printversion("procps", version("ps --version")) printversion("Python", version("python3 -V")) diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index f1936968703065..5af0d9045cbc39 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -10,6 +10,7 @@ # XZ-compressed kernel isn't currently supported on every listed arch. # # Arch Align Notes +# alpha 4 # arm 2/4 ARM and ARM-Thumb2 # arm64 4 # csky 2 @@ -44,6 +45,10 @@ BCJ= # Set the BCJ filter if one is available. # It must match the #ifdef usage in lib/decompress_unxz.c. case $SRCARCH in + alpha) + ALIGN=4 + ;; + arm) if is_enabled CONFIG_THUMB2_KERNEL; then ALIGN=2 diff --git a/security/Kconfig b/security/Kconfig index f7bf6cdc6229eb..4714274a552281 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -300,6 +300,7 @@ config SECURITY_COMMONCAP_KUNIT_TEST If unsure, say N. source "security/Kconfig.hardening" +source "security/Kconfig.openpax" endmenu diff --git a/security/Kconfig.openpax b/security/Kconfig.openpax new file mode 100644 index 00000000000000..df65b38ed396e4 --- /dev/null +++ b/security/Kconfig.openpax @@ -0,0 +1,90 @@ +# +# OpenPaX configuration +# + +menu "OpenPaX options" + +config OPENPAX + bool "Enable OpenPaX features" + default n + help + This configuration setting enables OpenPaX features. + OpenPaX adds memory safety-related defenses to the kernel which + reduce the risks posed by exploitable memory safety bugs. + +config OPENPAX_SOFTMODE + bool "Support PaX soft mode" + depends on OPENPAX + default y + help + Enabling this option will allow you to configure OpenPaX + features to run in soft mode. In this mode, OpenPaX features + will be disabled by default, only running on applications + which explicitly enable them. + + Soft mode can be enabled via the kernel.pax.softmode sysctl, + or the pax_softmode=1 kernel command-line option. + +config OPENPAX_XATTR_PAX_FLAGS + bool "Use filesystem extended attributes to modify OpenPaX features" + depends on OPENPAX + default y + help + Enabling this option will allow you to control whether + OpenPaX features are enabled on a per-executable basis via + xattr attributes. + + For compatibility with the original PaX patch, the feature + flags are read from the user.pax.flags extended attribute. + + If you disable this feature, then all applications will run + with OpenPaX enabled by default. + +config OPENPAX_MPROTECT + bool "Enforce W^X for memory mappings" + depends on OPENPAX + default y + help + Enabling this option prevents programs from making pages + executable when they are also writable. In addition, it + also denies transition of writable mappings to executable + mappings. + + This feature is known to break programs which depend on + just-in-time (JIT) compilation. It is advisable to enable + this feature system-wide, but mark programs which have + JIT compilation appropriately so the W^X enforcement is + disabled for them. + +config OPENPAX_EMUTRAMP + bool "Emulate stack and heap trampolines" + depends on OPENPAX + default y + help + Enabling this option allows programs to depend on common + types of stack and heap trampolines (such as the ones + generated by GCC and libffi) to continue working despite + the stack and heap being non-executable memory. + + This option works by intercepting the page faults caused + by executing code in non-executable memory and emulating + the side effects that would have happened from executing + the trampoline. + + Most likely, you should say 'y' here. + +config OPENPAX_EMUTRAMP_DEFAULT + bool "Enable trampoline emulation by default" + depends on OPENPAX_EMUTRAMP + default y + help + Enabling this option allows programs which require + trampolines to be emulated to continue working by default. + + Otherwise, the emulation flag must be enabled in a binary's + PaX marking, e.g. with paxmark -E . + + If you do not say 'y' here, you will have to manually mark + all programs which require trampoline emulation. + +endmenu diff --git a/security/apparmor/apparmorfs.c b/security/apparmor/apparmorfs.c index 5b42140e12e872..d1386722537d0f 100644 --- a/security/apparmor/apparmorfs.c +++ b/security/apparmor/apparmorfs.c @@ -2073,7 +2073,7 @@ int __aafs_profile_mkdir(struct aa_profile *profile, struct dentry *parent) return error; } -static struct dentry *ns_mkdir_op(struct mnt_idmap *idmap, struct inode *dir, +static struct dentry *ns_mkdir_op(const struct mnt_idmap *idmap, struct inode *dir, struct dentry *dentry, umode_t mode) { struct aa_ns *ns, *parent; diff --git a/security/apparmor/lsm.c b/security/apparmor/lsm.c index d502ad0ac26f5c..c73681d820a068 100644 --- a/security/apparmor/lsm.c +++ b/security/apparmor/lsm.c @@ -397,7 +397,7 @@ static int apparmor_path_rename(const struct path *old_dir, struct dentry *old_d label = begin_current_label_crit_section(&needput); if (!unconfined(label)) { - struct mnt_idmap *idmap = mnt_idmap(old_dir->mnt); + const struct mnt_idmap *idmap = mnt_idmap(old_dir->mnt); vfsuid_t vfsuid; struct path old_path = { .mnt = old_dir->mnt, .dentry = old_dentry }; @@ -485,7 +485,7 @@ static int apparmor_file_open(struct file *file) label = aa_get_newest_cred_label_condref(file->f_cred, &needput); if (!unconfined(label)) { - struct mnt_idmap *idmap = file_mnt_idmap(file); + const struct mnt_idmap *idmap = file_mnt_idmap(file); struct inode *inode = file_inode(file); vfsuid_t vfsuid; struct path_cond cond = { diff --git a/security/commoncap.c b/security/commoncap.c index 3399535808fec7..d47ab30223435b 100644 --- a/security/commoncap.c +++ b/security/commoncap.c @@ -348,7 +348,7 @@ int cap_inode_need_killpriv(struct dentry *dentry) * * Return: 0 if successful, -ve on error. */ -int cap_inode_killpriv(struct mnt_idmap *idmap, struct dentry *dentry) +int cap_inode_killpriv(const struct mnt_idmap *idmap, struct dentry *dentry) { int error; @@ -417,7 +417,7 @@ static bool is_v3header(int size, const struct vfs_cap_data *cap) * by the integrity subsystem, which really wants the unconverted values - * so that's good. */ -int cap_inode_getsecurity(struct mnt_idmap *idmap, +int cap_inode_getsecurity(const struct mnt_idmap *idmap, struct inode *inode, const char *name, void **buffer, bool alloc) { @@ -566,7 +566,7 @@ static bool validheader(size_t size, const struct vfs_cap_data *cap) * * Return: On success, return the new size; on error, return < 0. */ -int cap_convert_nscap(struct mnt_idmap *idmap, struct dentry *dentry, +int cap_convert_nscap(const struct mnt_idmap *idmap, struct dentry *dentry, const void **ivalue, size_t size) { struct vfs_ns_cap_data *nscap; @@ -672,7 +672,7 @@ static inline int bprm_caps_from_vfs_caps(struct cpu_vfs_cap_data *caps, * permissions. On non-idmapped mounts or if permission checking is to be * performed on the raw inode simply pass @nop_mnt_idmap. */ -int get_vfs_caps_from_disk(struct mnt_idmap *idmap, +int get_vfs_caps_from_disk(const struct mnt_idmap *idmap, const struct dentry *dentry, struct cpu_vfs_cap_data *cpu_caps) { @@ -1063,7 +1063,7 @@ int cap_inode_setxattr(struct dentry *dentry, const char *name, * This is used to make sure security xattrs don't get removed by those who * aren't privileged to remove them. */ -int cap_inode_removexattr(struct mnt_idmap *idmap, +int cap_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name) { struct user_namespace *user_ns = dentry->d_sb->s_user_ns; diff --git a/security/integrity/evm/evm_main.c b/security/integrity/evm/evm_main.c index b59e3f121b8a89..9bfea858df2d3b 100644 --- a/security/integrity/evm/evm_main.c +++ b/security/integrity/evm/evm_main.c @@ -481,7 +481,7 @@ static enum integrity_status evm_verify_current_integrity(struct dentry *dentry) * * Returns 1 if passed xattr value differs from current value, 0 otherwise. */ -static int evm_xattr_change(struct mnt_idmap *idmap, +static int evm_xattr_change(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name, const void *xattr_value, size_t xattr_value_len) { @@ -517,7 +517,7 @@ static int evm_xattr_change(struct mnt_idmap *idmap, * For posix xattr acls only, permit security.evm, even if it currently * doesn't exist, to be updated unless the EVM signature is immutable. */ -static int evm_protect_xattr(struct mnt_idmap *idmap, +static int evm_protect_xattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name, const void *xattr_value, size_t xattr_value_len) { @@ -607,7 +607,7 @@ static int evm_protect_xattr(struct mnt_idmap *idmap, * userspace from writing HMAC value. Writing 'security.evm' requires * requires CAP_SYS_ADMIN privileges. */ -static int evm_inode_setxattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int evm_inode_setxattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name, const void *xattr_value, size_t xattr_value_len, int flags) { @@ -639,7 +639,7 @@ static int evm_inode_setxattr(struct mnt_idmap *idmap, struct dentry *dentry, * Removing 'security.evm' requires CAP_SYS_ADMIN privileges and that * the current value is valid. */ -static int evm_inode_removexattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int evm_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name) { /* Policy permits modification of the protected xattrs even though @@ -652,7 +652,7 @@ static int evm_inode_removexattr(struct mnt_idmap *idmap, struct dentry *dentry, } #ifdef CONFIG_FS_POSIX_ACL -static int evm_inode_set_acl_change(struct mnt_idmap *idmap, +static int evm_inode_set_acl_change(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name, struct posix_acl *kacl) { @@ -671,7 +671,7 @@ static int evm_inode_set_acl_change(struct mnt_idmap *idmap, return 0; } #else -static inline int evm_inode_set_acl_change(struct mnt_idmap *idmap, +static inline int evm_inode_set_acl_change(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name, struct posix_acl *kacl) @@ -693,7 +693,7 @@ static inline int evm_inode_set_acl_change(struct mnt_idmap *idmap, * * Return: zero on success, -EPERM on failure. */ -static int evm_inode_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, +static int evm_inode_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name, struct posix_acl *kacl) { enum integrity_status evm_status; @@ -745,7 +745,7 @@ static int evm_inode_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, * * Return: zero on success, -EPERM on failure. */ -static int evm_inode_remove_acl(struct mnt_idmap *idmap, struct dentry *dentry, +static int evm_inode_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { return evm_inode_set_acl(idmap, dentry, acl_name, NULL); @@ -926,14 +926,14 @@ static void evm_inode_post_removexattr(struct dentry *dentry, * Update the 'security.evm' xattr with the EVM HMAC re-calculated after * removing posix acls. */ -static inline void evm_inode_post_remove_acl(struct mnt_idmap *idmap, +static inline void evm_inode_post_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { evm_inode_post_removexattr(dentry, acl_name); } -static int evm_attr_change(struct mnt_idmap *idmap, +static int evm_attr_change(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *attr) { struct inode *inode = d_backing_inode(dentry); @@ -956,7 +956,7 @@ static int evm_attr_change(struct mnt_idmap *idmap, * Permit update of file attributes when files have a valid EVM signature, * except in the case of them having an immutable portable signature. */ -static int evm_inode_setattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int evm_inode_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *attr) { unsigned int ia_valid = attr->ia_valid; @@ -1008,7 +1008,7 @@ static int evm_inode_setattr(struct mnt_idmap *idmap, struct dentry *dentry, * This function is called from notify_change(), which expects the caller * to lock the inode's i_mutex. */ -static void evm_inode_post_setattr(struct mnt_idmap *idmap, +static void evm_inode_post_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, int ia_valid) { if (!evm_revalidate_status(NULL)) @@ -1140,7 +1140,7 @@ static void evm_file_release(struct file *file) iint->flags &= ~EVM_NEW_FILE; } -static void evm_post_path_mknod(struct mnt_idmap *idmap, struct dentry *dentry) +static void evm_post_path_mknod(const struct mnt_idmap *idmap, struct dentry *dentry) { struct inode *inode = d_backing_inode(dentry); struct evm_iint_cache *iint = evm_iint_inode(inode); diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h index 10214f73ca1ed9..b502854f28ee71 100644 --- a/security/integrity/ima/ima.h +++ b/security/integrity/ima/ima.h @@ -423,7 +423,7 @@ static inline void ima_process_queued_keys(void) {} #endif /* CONFIG_IMA_QUEUE_EARLY_BOOT_KEYS */ /* LIM API function definitions */ -int ima_get_action(struct mnt_idmap *idmap, struct inode *inode, +int ima_get_action(const struct mnt_idmap *idmap, struct inode *inode, const struct cred *cred, struct lsm_prop *prop, int mask, enum ima_hooks func, int *pcr, struct ima_template_desc **template_desc, @@ -437,7 +437,7 @@ void ima_store_measurement(struct ima_iint_cache *iint, struct file *file, struct evm_ima_xattr_data *xattr_value, int xattr_len, const struct modsig *modsig, int pcr, struct ima_template_desc *template_desc); -int process_buffer_measurement(struct mnt_idmap *idmap, +int process_buffer_measurement(const struct mnt_idmap *idmap, struct inode *inode, const void *buf, int size, const char *eventname, enum ima_hooks func, int pcr, const char *func_data, @@ -454,7 +454,7 @@ void ima_free_template_entry(struct ima_template_entry *entry); const char *ima_d_path(const struct path *path, char **pathbuf, char *filename); /* IMA policy related functions */ -int ima_match_policy(struct mnt_idmap *idmap, struct inode *inode, +int ima_match_policy(const struct mnt_idmap *idmap, struct inode *inode, const struct cred *cred, struct lsm_prop *prop, enum ima_hooks func, int mask, int flags, int *pcr, struct ima_template_desc **template_desc, @@ -489,7 +489,7 @@ int ima_appraise_measurement(enum ima_hooks func, struct ima_iint_cache *iint, struct evm_ima_xattr_data *xattr_value, int xattr_len, const struct modsig *modsig, bool bprm_is_check); -int ima_must_appraise(struct mnt_idmap *idmap, struct inode *inode, +int ima_must_appraise(const struct mnt_idmap *idmap, struct inode *inode, int mask, enum ima_hooks func); void ima_update_xattr(struct ima_iint_cache *iint, struct file *file); enum integrity_status ima_get_cache_status(struct ima_iint_cache *iint, @@ -519,7 +519,7 @@ static inline int ima_appraise_measurement(enum ima_hooks func, return INTEGRITY_UNKNOWN; } -static inline int ima_must_appraise(struct mnt_idmap *idmap, +static inline int ima_must_appraise(const struct mnt_idmap *idmap, struct inode *inode, int mask, enum ima_hooks func) { diff --git a/security/integrity/ima/ima_api.c b/security/integrity/ima/ima_api.c index 122d127e108dcc..3c5a23b4a2ffc0 100644 --- a/security/integrity/ima/ima_api.c +++ b/security/integrity/ima/ima_api.c @@ -188,7 +188,7 @@ void ima_add_violation(struct file *file, const unsigned char *filename, * Returns IMA_MEASURE, IMA_APPRAISE mask. * */ -int ima_get_action(struct mnt_idmap *idmap, struct inode *inode, +int ima_get_action(const struct mnt_idmap *idmap, struct inode *inode, const struct cred *cred, struct lsm_prop *prop, int mask, enum ima_hooks func, int *pcr, struct ima_template_desc **template_desc, diff --git a/security/integrity/ima/ima_appraise.c b/security/integrity/ima/ima_appraise.c index b280488e15fcb0..63b0cb957c6528 100644 --- a/security/integrity/ima/ima_appraise.c +++ b/security/integrity/ima/ima_appraise.c @@ -71,7 +71,7 @@ bool is_ima_appraise_enabled(void) * * Return 1 to appraise or hash */ -int ima_must_appraise(struct mnt_idmap *idmap, struct inode *inode, +int ima_must_appraise(const struct mnt_idmap *idmap, struct inode *inode, int mask, enum ima_hooks func) { struct lsm_prop prop; @@ -634,7 +634,7 @@ void ima_update_xattr(struct ima_iint_cache *iint, struct file *file) * This function is called from notify_change(), which expects the caller * to lock the inode's i_mutex. */ -static void ima_inode_post_setattr(struct mnt_idmap *idmap, +static void ima_inode_post_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, int ia_valid) { struct inode *inode = d_backing_inode(dentry); @@ -759,7 +759,7 @@ static int validate_hash_algo(struct dentry *dentry, return -EACCES; } -static int ima_inode_setxattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int ima_inode_setxattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name, const void *xattr_value, size_t xattr_value_len, int flags) { @@ -792,7 +792,7 @@ static int ima_inode_setxattr(struct mnt_idmap *idmap, struct dentry *dentry, return result; } -static int ima_inode_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, +static int ima_inode_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name, struct posix_acl *kacl) { if (evm_revalidate_status(acl_name)) @@ -801,7 +801,7 @@ static int ima_inode_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, return 0; } -static int ima_inode_removexattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int ima_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *xattr_name) { int result, digsig = -1; @@ -817,7 +817,7 @@ static int ima_inode_removexattr(struct mnt_idmap *idmap, struct dentry *dentry, return result; } -static int ima_inode_remove_acl(struct mnt_idmap *idmap, struct dentry *dentry, +static int ima_inode_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { return ima_inode_set_acl(idmap, dentry, acl_name, NULL); diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c index ab1e53b3210d3a..7ac38a98b1f961 100644 --- a/security/integrity/ima/ima_main.c +++ b/security/integrity/ima/ima_main.c @@ -846,7 +846,7 @@ EXPORT_SYMBOL_GPL(ima_inode_hash); * Skip calling process_measurement(), but indicate which newly, created * tmpfiles are in policy. */ -static void ima_post_create_tmpfile(struct mnt_idmap *idmap, +static void ima_post_create_tmpfile(const struct mnt_idmap *idmap, struct inode *inode) { @@ -879,7 +879,7 @@ static void ima_post_create_tmpfile(struct mnt_idmap *idmap, * Mark files created via the mknodat syscall as new, so that the * file data can be written later. */ -static void ima_post_path_mknod(struct mnt_idmap *idmap, struct dentry *dentry) +static void ima_post_path_mknod(const struct mnt_idmap *idmap, struct dentry *dentry) { struct ima_iint_cache *iint; struct inode *inode = dentry->d_inode; @@ -1095,7 +1095,7 @@ static int ima_post_load_data(char *buf, loff_t size, * has been written to the passed location but not added to a measurement entry, * a negative value otherwise. */ -int process_buffer_measurement(struct mnt_idmap *idmap, +int process_buffer_measurement(const struct mnt_idmap *idmap, struct inode *inode, const void *buf, int size, const char *eventname, enum ima_hooks func, int pcr, const char *func_data, diff --git a/security/integrity/ima/ima_policy.c b/security/integrity/ima/ima_policy.c index 68d9a5e6c232ea..b34d9621929bf1 100644 --- a/security/integrity/ima/ima_policy.c +++ b/security/integrity/ima/ima_policy.c @@ -580,7 +580,7 @@ static bool ima_match_rule_data(struct ima_rule_entry *rule, * Returns true on rule match, false on failure. */ static bool ima_match_rules(struct ima_rule_entry *rule, - struct mnt_idmap *idmap, + const struct mnt_idmap *idmap, struct inode *inode, const struct cred *cred, struct lsm_prop *prop, enum ima_hooks func, int mask, const char *func_data) @@ -762,7 +762,7 @@ static int get_subaction(struct ima_rule_entry *rule, enum ima_hooks func) * list when walking it. Reads are many orders of magnitude more numerous * than writes so ima_match_policy() is classical RCU candidate. */ -int ima_match_policy(struct mnt_idmap *idmap, struct inode *inode, +int ima_match_policy(const struct mnt_idmap *idmap, struct inode *inode, const struct cred *cred, struct lsm_prop *prop, enum ima_hooks func, int mask, int flags, int *pcr, struct ima_template_desc **template_desc, diff --git a/security/keys/trusted-keys/trusted_tpm1.c b/security/keys/trusted-keys/trusted_tpm1.c index bf0bf7f3697052..e5a904b5c19469 100644 --- a/security/keys/trusted-keys/trusted_tpm1.c +++ b/security/keys/trusted-keys/trusted_tpm1.c @@ -101,7 +101,7 @@ static inline void dump_tpm_buf(unsigned char *buf) static int TSS_rawhmac(unsigned char *digest, const unsigned char *key, unsigned int keylen, ...) { - struct hmac_sha1_ctx hmac_ctx; + struct hmac_sha1_ctx hmac_ctx __cleanup(hmac_sha1_zeroize_ctx); va_list argp; unsigned int dlen; unsigned char *data; diff --git a/security/lsm_audit.c b/security/lsm_audit.c index 737f5a263a8f79..955b2e7b2c8e4e 100644 --- a/security/lsm_audit.c +++ b/security/lsm_audit.c @@ -182,7 +182,7 @@ void audit_log_lsm_data(struct audit_buffer *ab, * start making this union too large! See struct lsm_network_audit * as an example of how to deal with large data. */ - BUILD_BUG_ON(sizeof(a->u) > sizeof(void *)*2); + BUILD_BUG_ON(sizeof(a->u) > (sizeof(u64) * 2)); switch (a->type) { case LSM_AUDIT_DATA_NONE: @@ -231,7 +231,7 @@ void audit_log_lsm_data(struct audit_buffer *ab, audit_log_format(ab, " ino=%llu", inode->i_ino); } - audit_log_format(ab, " ioctlcmd=0x%hx", a->u.op->cmd); + audit_log_format(ab, " ioctlcmd=0x%x", a->u.op->cmd); break; } case LSM_AUDIT_DATA_DENTRY: { @@ -403,6 +403,10 @@ void audit_log_lsm_data(struct audit_buffer *ab, case LSM_AUDIT_DATA_NLMSGTYPE: audit_log_format(ab, " nl-msgtype=%hu", a->u.nlmsg_type); break; + case LSM_AUDIT_DATA_NS: + audit_log_format(ab, " namespace_type=0x%x namespace_id=%llu", + a->u.ns.ns_type, a->u.ns.ns_id); + break; } /* switch (a->type) */ } diff --git a/security/lsm_init.c b/security/lsm_init.c index a1ad641811de06..8d5ecea99c46b8 100644 --- a/security/lsm_init.c +++ b/security/lsm_init.c @@ -290,7 +290,6 @@ static void __init lsm_prepare(struct lsm_info *lsm) return; /* Register the LSM blob sizes. */ - blobs = lsm->blobs; lsm_blob_size_update(&blobs->lbs_cred, &blob_sizes.lbs_cred); lsm_blob_size_update(&blobs->lbs_file, &blob_sizes.lbs_file); lsm_blob_size_update(&blobs->lbs_backing_file, @@ -303,6 +302,7 @@ static void __init lsm_prepare(struct lsm_info *lsm) lsm_blob_size_update(&blobs->lbs_ipc, &blob_sizes.lbs_ipc); lsm_blob_size_update(&blobs->lbs_key, &blob_sizes.lbs_key); lsm_blob_size_update(&blobs->lbs_msg_msg, &blob_sizes.lbs_msg_msg); + lsm_blob_size_update(&blobs->lbs_ns, &blob_sizes.lbs_ns); lsm_blob_size_update(&blobs->lbs_perf_event, &blob_sizes.lbs_perf_event); lsm_blob_size_update(&blobs->lbs_sock, &blob_sizes.lbs_sock); @@ -431,8 +431,7 @@ int __init security_init(void) } if (lsm_order_cmdline) { - if (lsm_order_legacy) - lsm_order_legacy = NULL; + lsm_order_legacy = NULL; lsm_order_parse(lsm_order_cmdline, "cmdline"); } else lsm_order_parse(lsm_order_builtin, "builtin"); @@ -450,6 +449,7 @@ int __init security_init(void) lsm_pr("blob(ipc) size %d\n", blob_sizes.lbs_ipc); lsm_pr("blob(key) size %d\n", blob_sizes.lbs_key); lsm_pr("blob(msg_msg)_size %d\n", blob_sizes.lbs_msg_msg); + lsm_pr("blob(ns) size %d\n", blob_sizes.lbs_ns); lsm_pr("blob(sock) size %d\n", blob_sizes.lbs_sock); lsm_pr("blob(superblock) size %d\n", blob_sizes.lbs_superblock); lsm_pr("blob(perf_event) size %d\n", blob_sizes.lbs_perf_event); @@ -476,8 +476,7 @@ int __init security_init(void) blob_sizes.lbs_inode, 0, SLAB_PANIC, NULL); - if (lsm_cred_alloc((struct cred *)unrcu_pointer(current->cred), - GFP_KERNEL)) + if (lsm_cred_alloc((struct cred *)current->cred, GFP_KERNEL)) panic("early LSM cred alloc failed\n"); if (lsm_task_alloc(current)) panic("early LSM task alloc failed\n"); diff --git a/security/security.c b/security/security.c index 2ee276ab15c5bc..f6583d50739592 100644 --- a/security/security.c +++ b/security/security.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -381,6 +382,19 @@ static int lsm_superblock_alloc(struct super_block *sb) GFP_KERNEL); } +/** + * lsm_ns_alloc - allocate a composite namespace blob + * @ns: the namespace that needs a blob + * + * Allocate the namespace blob for all the modules + * + * Returns 0, or -ENOMEM if memory can't be allocated. + */ +static int lsm_ns_alloc(struct ns_common *ns) +{ + return lsm_blob_alloc(&ns->ns_security, blob_sizes.lbs_ns, GFP_KERNEL); +} + /** * lsm_fill_user_ctx - Fill a user space lsm_ctx structure * @uctx: a userspace LSM context to be filled @@ -1425,7 +1439,7 @@ EXPORT_SYMBOL(security_path_mknod); * * Update inode security field after a regular file has been created. */ -void security_path_post_mknod(struct mnt_idmap *idmap, struct dentry *dentry) +void security_path_post_mknod(const struct mnt_idmap *idmap, struct dentry *dentry) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) return; @@ -1614,6 +1628,7 @@ int security_path_chroot(const struct path *path) /** * security_inode_create() - Check if creating a file is allowed + * @idmap: idmap of the mount * @dir: the parent directory * @dentry: the file being created * @mode: requested file mode @@ -1622,12 +1637,12 @@ int security_path_chroot(const struct path *path) * * Return: Returns 0 if permission is granted. */ -int security_inode_create(struct inode *dir, struct dentry *dentry, - umode_t mode) +int security_inode_create(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode) { if (unlikely(IS_PRIVATE(dir))) return 0; - return call_int_hook(inode_create, dir, dentry, mode); + return call_int_hook(inode_create, idmap, dir, dentry, mode); } EXPORT_SYMBOL_GPL(security_inode_create); @@ -1638,7 +1653,7 @@ EXPORT_SYMBOL_GPL(security_inode_create); * * Update inode security data after a tmpfile has been created. */ -void security_inode_post_create_tmpfile(struct mnt_idmap *idmap, +void security_inode_post_create_tmpfile(const struct mnt_idmap *idmap, struct inode *inode) { if (unlikely(IS_PRIVATE(inode))) @@ -1648,6 +1663,7 @@ void security_inode_post_create_tmpfile(struct mnt_idmap *idmap, /** * security_inode_link() - Check if creating a hard link is allowed + * @idmap: idmap of the mount * @old_dentry: existing file * @dir: new parent directory * @new_dentry: new link @@ -1656,12 +1672,12 @@ void security_inode_post_create_tmpfile(struct mnt_idmap *idmap, * * Return: Returns 0 if permission is granted. */ -int security_inode_link(struct dentry *old_dentry, struct inode *dir, - struct dentry *new_dentry) +int security_inode_link(const struct mnt_idmap *idmap, struct dentry *old_dentry, + struct inode *dir, struct dentry *new_dentry) { if (unlikely(IS_PRIVATE(d_backing_inode(old_dentry)))) return 0; - return call_int_hook(inode_link, old_dentry, dir, new_dentry); + return call_int_hook(inode_link, idmap, old_dentry, dir, new_dentry); } /** @@ -1682,6 +1698,7 @@ int security_inode_unlink(struct inode *dir, struct dentry *dentry) /** * security_inode_symlink() - Check if creating a symbolic link is allowed + * @idmap: idmap of the mount * @dir: parent directory * @dentry: symbolic link * @old_name: existing filename @@ -1690,16 +1707,17 @@ int security_inode_unlink(struct inode *dir, struct dentry *dentry) * * Return: Returns 0 if permission is granted. */ -int security_inode_symlink(struct inode *dir, struct dentry *dentry, - const char *old_name) +int security_inode_symlink(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, const char *old_name) { if (unlikely(IS_PRIVATE(dir))) return 0; - return call_int_hook(inode_symlink, dir, dentry, old_name); + return call_int_hook(inode_symlink, idmap, dir, dentry, old_name); } /** * security_inode_mkdir() - Check if creating a new directory is allowed + * @idmap: idmap of the mount * @dir: parent directory * @dentry: new directory * @mode: new directory mode @@ -1709,11 +1727,12 @@ int security_inode_symlink(struct inode *dir, struct dentry *dentry, * * Return: Returns 0 if permission is granted. */ -int security_inode_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode) +int security_inode_mkdir(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode) { if (unlikely(IS_PRIVATE(dir))) return 0; - return call_int_hook(inode_mkdir, dir, dentry, mode); + return call_int_hook(inode_mkdir, idmap, dir, dentry, mode); } EXPORT_SYMBOL_GPL(security_inode_mkdir); @@ -1735,6 +1754,7 @@ int security_inode_rmdir(struct inode *dir, struct dentry *dentry) /** * security_inode_mknod() - Check if creating a special file is allowed + * @idmap: idmap of the mount * @dir: parent directory * @dentry: new file * @mode: new file mode @@ -1747,12 +1767,12 @@ int security_inode_rmdir(struct inode *dir, struct dentry *dentry) * * Return: Returns 0 if permission is granted. */ -int security_inode_mknod(struct inode *dir, struct dentry *dentry, - umode_t mode, dev_t dev) +int security_inode_mknod(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode, dev_t dev) { if (unlikely(IS_PRIVATE(dir))) return 0; - return call_int_hook(inode_mknod, dir, dentry, mode, dev); + return call_int_hook(inode_mknod, idmap, dir, dentry, mode, dev); } /** @@ -1823,6 +1843,7 @@ int security_inode_follow_link(struct dentry *dentry, struct inode *inode, /** * security_inode_permission() - Check if accessing an inode is allowed + * @idmap: idmap of the mount * @inode: inode * @mask: access mask * @@ -1835,11 +1856,12 @@ int security_inode_follow_link(struct dentry *dentry, struct inode *inode, * * Return: Returns 0 if permission is granted. */ -int security_inode_permission(struct inode *inode, int mask) +int security_inode_permission(const struct mnt_idmap *idmap, + struct inode *inode, int mask) { if (unlikely(IS_PRIVATE(inode))) return 0; - return call_int_hook(inode_permission, inode, mask); + return call_int_hook(inode_permission, idmap, inode, mask); } /** @@ -1855,7 +1877,7 @@ int security_inode_permission(struct inode *inode, int mask) * * Return: Returns 0 if permission is granted. */ -int security_inode_setattr(struct mnt_idmap *idmap, +int security_inode_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *attr) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) @@ -1872,7 +1894,7 @@ EXPORT_SYMBOL_GPL(security_inode_setattr); * * Update inode security field after successful setting file attributes. */ -void security_inode_post_setattr(struct mnt_idmap *idmap, struct dentry *dentry, +void security_inode_post_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, int ia_valid) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) @@ -1921,7 +1943,7 @@ int security_inode_getattr(const struct path *path) * * Return: Returns 0 if permission is granted. */ -int security_inode_setxattr(struct mnt_idmap *idmap, +int security_inode_setxattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name, const void *value, size_t size, int flags) { @@ -1953,7 +1975,7 @@ int security_inode_setxattr(struct mnt_idmap *idmap, * * Return: Returns 0 if permission is granted. */ -int security_inode_set_acl(struct mnt_idmap *idmap, +int security_inode_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name, struct posix_acl *kacl) { @@ -1990,7 +2012,7 @@ void security_inode_post_set_acl(struct dentry *dentry, const char *acl_name, * * Return: Returns 0 if permission is granted. */ -int security_inode_get_acl(struct mnt_idmap *idmap, +int security_inode_get_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) @@ -2009,7 +2031,7 @@ int security_inode_get_acl(struct mnt_idmap *idmap, * * Return: Returns 0 if permission is granted. */ -int security_inode_remove_acl(struct mnt_idmap *idmap, +int security_inode_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) @@ -2026,7 +2048,7 @@ int security_inode_remove_acl(struct mnt_idmap *idmap, * Update inode security data after successfully removing posix acls on * @dentry in @idmap. The posix acls are identified by @acl_name. */ -void security_inode_post_remove_acl(struct mnt_idmap *idmap, +void security_inode_post_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { if (unlikely(IS_PRIVATE(d_backing_inode(dentry)))) @@ -2108,7 +2130,7 @@ int security_inode_listxattr(struct dentry *dentry) * * Return: Returns 0 if permission is granted. */ -int security_inode_removexattr(struct mnt_idmap *idmap, +int security_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name) { int rc; @@ -2197,7 +2219,7 @@ int security_inode_need_killpriv(struct dentry *dentry) * Return: Return 0 on success. If error is returned, then the operation * causing setuid bit removal is failed. */ -int security_inode_killpriv(struct mnt_idmap *idmap, +int security_inode_killpriv(const struct mnt_idmap *idmap, struct dentry *dentry) { return call_int_hook(inode_killpriv, idmap, dentry); @@ -2219,7 +2241,7 @@ int security_inode_killpriv(struct mnt_idmap *idmap, * * Return: Returns size of buffer on success. */ -int security_inode_getsecurity(struct mnt_idmap *idmap, +int security_inode_getsecurity(const struct mnt_idmap *idmap, struct inode *inode, const char *name, void **buffer, bool alloc) { @@ -2486,9 +2508,8 @@ void security_backing_file_free(struct file *backing_file) { void *blob = backing_file_security(backing_file); - call_void_hook(backing_file_free, backing_file); - if (blob) { + call_void_hook(backing_file_free, backing_file); backing_file_set_security(backing_file, NULL); kmem_cache_free(lsm_backing_file_cache, blob); } @@ -3357,6 +3378,64 @@ int security_create_user_ns(const struct cred *cred) return call_int_hook(userns_create, cred); } +/** + * security_namespace_init() - Initialize LSM security data for a namespace + * @ns: the namespace being initialized + * + * Initialize the LSM security blob attached to the namespace. The namespace type + * is available via ns->ns_type, and the owning user namespace (if any) + * via ns->ops->owner(ns). + * + * Return: Returns 0 if successful, otherwise < 0 error code. + */ +int security_namespace_init(struct ns_common *ns) +{ + int rc; + + rc = lsm_ns_alloc(ns); + if (unlikely(rc)) + return rc; + + rc = call_int_hook(namespace_init, ns); + if (unlikely(rc)) + security_namespace_free(ns); + + return rc; +} + +/** + * security_namespace_free() - Release LSM security data from a namespace + * @ns: the namespace being freed + * + * Release security data attached to the namespace. Called before the + * namespace structure is freed. + */ +void security_namespace_free(struct ns_common *ns) +{ + if (!ns->ns_security) + return; + + call_void_hook(namespace_free, ns); + + kfree(ns->ns_security); + ns->ns_security = NULL; +} + +/** + * security_namespace_install() - Check permission to install a namespace + * @nsset: the target nsset being configured + * @ns: the namespace being installed + * + * Check permission before allowing a namespace to be installed into the + * process's set of namespaces via setns(2). + * + * Return: Returns 0 if permission is granted, otherwise < 0 error code. + */ +int security_namespace_install(const struct nsset *nsset, struct ns_common *ns) +{ + return call_int_hook(namespace_install, nsset, ns); +} + /** * security_ipc_permission() - Check if sysv ipc access is allowed * @ipcp: ipc permission structure diff --git a/security/selinux/avc.c b/security/selinux/avc.c index b91f914d292e58..2a9d4888384b2c 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -190,8 +190,8 @@ avc_xperms_decision_lookup(u8 driver, u8 base_perm, } static inline unsigned int -avc_xperms_has_perm(struct extended_perms_decision *xpd, - u8 perm, u8 which) +avc_xperms_has_perm(const struct extended_perms_decision *xpd, + u8 perm, u8 which) { unsigned int rc = 0; @@ -247,7 +247,7 @@ static void avc_xperms_free(struct avc_xperms_node *xp_node) } static void avc_copy_xperms_decision(struct extended_perms_decision *dest, - struct extended_perms_decision *src) + const struct extended_perms_decision *src) { dest->base_perm = src->base_perm; dest->driver = src->driver; @@ -269,7 +269,7 @@ static void avc_copy_xperms_decision(struct extended_perms_decision *dest, */ static inline void avc_quick_copy_xperms_decision(u8 perm, struct extended_perms_decision *dest, - struct extended_perms_decision *src) + const struct extended_perms_decision *src) { /* * compute index of the u32 of the 256 bits (8 u32s) that contain this @@ -323,7 +323,7 @@ static struct avc_xperms_decision_node } static int avc_add_xperms_decision(struct avc_node *node, - struct extended_perms_decision *src) + const struct extended_perms_decision *src) { struct avc_xperms_decision_node *dest_xpd; @@ -348,7 +348,7 @@ static struct avc_xperms_node *avc_xperms_alloc(void) } static int avc_xperms_populate(struct avc_node *node, - struct avc_xperms_node *src) + const struct avc_xperms_node *src) { struct avc_xperms_node *dest; struct avc_xperms_decision_node *dest_xpd; @@ -381,14 +381,17 @@ static int avc_xperms_populate(struct avc_node *node, } static inline u32 avc_xperms_audit_required(u32 requested, - struct av_decision *avd, - struct extended_perms_decision *xpd, - u8 perm, - int result, - u32 *deniedp) + const struct av_decision *avd, + const struct extended_perms_decision *xpd, + u8 perm, + int result, + u32 *deniedp) { u32 denied, audited; + if (avd->flags & AVD_FLAGS_NEVERAUDIT) + return 0; + denied = requested & ~avd->allowed; if (unlikely(denied)) { audited = denied & avd->auditdeny; @@ -411,8 +414,8 @@ static inline u32 avc_xperms_audit_required(u32 requested, } static inline int avc_xperms_audit(u32 ssid, u32 tsid, u16 tclass, - u32 requested, struct av_decision *avd, - struct extended_perms_decision *xpd, + u32 requested, const struct av_decision *avd, + const struct extended_perms_decision *xpd, u8 perm, int result, struct common_audit_data *ad) { @@ -509,7 +512,8 @@ static struct avc_node *avc_alloc_node(void) return node; } -static void avc_node_populate(struct avc_node *node, u32 ssid, u32 tsid, u16 tclass, struct av_decision *avd) +static void avc_node_populate(struct avc_node *node, u32 ssid, u32 tsid, + u16 tclass, const struct av_decision *avd) { node->ae.ssid = ssid; node->ae.tsid = tsid; @@ -603,7 +607,8 @@ static int avc_latest_notif_update(u32 seqno, int is_insert) * the access vectors into a cache entry. */ static void avc_insert(u32 ssid, u32 tsid, u16 tclass, - struct av_decision *avd, struct avc_xperms_node *xp_node) + const struct av_decision *avd, + const struct avc_xperms_node *xp_node) { struct avc_node *pos, *node = NULL; u32 hvalue; @@ -828,7 +833,7 @@ int __init avc_add_callback(int (*callback)(u32 event), u32 events) */ static int avc_update_node(u32 event, u32 perms, u8 driver, u8 base_perm, u8 xperm, u32 ssid, u32 tsid, u16 tclass, u32 seqno, - struct extended_perms_decision *xpd, u32 flags) + const struct extended_perms_decision *xpd, u32 flags) { u32 hvalue; int rc = 0; diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 30776b0107f291..6168dd5f91c227 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -3087,12 +3087,14 @@ static int selinux_inode_init_security_anon(struct inode *inode, &ad); } -static int selinux_inode_create(struct inode *dir, struct dentry *dentry, umode_t mode) +static int selinux_inode_create(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode) { return may_create(dir, dentry, SECCLASS_FILE); } -static int selinux_inode_link(struct dentry *old_dentry, struct inode *dir, struct dentry *new_dentry) +static int selinux_inode_link(const struct mnt_idmap *idmap, struct dentry *old_dentry, + struct inode *dir, struct dentry *new_dentry) { return may_link(dir, old_dentry, MAY_LINK); } @@ -3102,12 +3104,14 @@ static int selinux_inode_unlink(struct inode *dir, struct dentry *dentry) return may_link(dir, dentry, MAY_UNLINK); } -static int selinux_inode_symlink(struct inode *dir, struct dentry *dentry, const char *name) +static int selinux_inode_symlink(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, const char *name) { return may_create(dir, dentry, SECCLASS_LNK_FILE); } -static int selinux_inode_mkdir(struct inode *dir, struct dentry *dentry, umode_t mask) +static int selinux_inode_mkdir(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mask) { return may_create(dir, dentry, SECCLASS_DIR); } @@ -3117,7 +3121,8 @@ static int selinux_inode_rmdir(struct inode *dir, struct dentry *dentry) return may_link(dir, dentry, MAY_RMDIR); } -static int selinux_inode_mknod(struct inode *dir, struct dentry *dentry, umode_t mode, dev_t dev) +static int selinux_inode_mknod(const struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode, dev_t dev) { return may_create(dir, dentry, inode_mode_to_security_class(mode)); } @@ -3249,13 +3254,15 @@ static inline void task_avdcache_update(struct task_security_struct *tsec, /** * selinux_inode_permission - Check if the current task can access an inode + * @idmap: idmap of the mount * @inode: the inode that is being accessed * @requested: the accesses being requested * * Check if the current task is allowed to access @inode according to * @requested. Returns 0 if allowed, negative values otherwise. */ -static int selinux_inode_permission(struct inode *inode, int requested) +static int selinux_inode_permission(const struct mnt_idmap *idmap, + struct inode *inode, int requested) { int mask; u32 perms; @@ -3310,7 +3317,7 @@ static int selinux_inode_permission(struct inode *inode, int requested) return rc; } -static int selinux_inode_setattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int selinux_inode_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *iattr) { const struct cred *cred = current_cred(); @@ -3380,7 +3387,7 @@ static int selinux_inode_xattr_skipcap(const char *name) return !strcmp(name, XATTR_NAME_SELINUX); } -static int selinux_inode_setxattr(struct mnt_idmap *idmap, +static int selinux_inode_setxattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name, const void *value, size_t size, int flags) { @@ -3466,20 +3473,20 @@ static int selinux_inode_setxattr(struct mnt_idmap *idmap, &ad); } -static int selinux_inode_set_acl(struct mnt_idmap *idmap, +static int selinux_inode_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name, struct posix_acl *kacl) { return dentry_has_perm(current_cred(), dentry, FILE__SETATTR); } -static int selinux_inode_get_acl(struct mnt_idmap *idmap, +static int selinux_inode_get_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { return dentry_has_perm(current_cred(), dentry, FILE__GETATTR); } -static int selinux_inode_remove_acl(struct mnt_idmap *idmap, +static int selinux_inode_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { return dentry_has_perm(current_cred(), dentry, FILE__SETATTR); @@ -3539,7 +3546,7 @@ static int selinux_inode_listxattr(struct dentry *dentry) return dentry_has_perm(cred, dentry, FILE__GETATTR); } -static int selinux_inode_removexattr(struct mnt_idmap *idmap, +static int selinux_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name) { /* if not a selinux xattr, only check the ordinary setattr perm */ @@ -3619,7 +3626,7 @@ static int selinux_path_notify(const struct path *path, u64 mask, * * Permission check is handled by selinux_inode_getxattr hook. */ -static int selinux_inode_getsecurity(struct mnt_idmap *idmap, +static int selinux_inode_getsecurity(const struct mnt_idmap *idmap, struct inode *inode, const char *name, void **buffer, bool alloc) { @@ -3929,7 +3936,7 @@ static void selinux_backing_file_free(struct file *backing_file) * operation to an inode. */ static int ioctl_has_perm(const struct cred *cred, struct file *file, - u32 requested, u16 cmd) + u32 requested, unsigned int cmd) { struct common_audit_data ad; struct file_security_struct *fsec = selinux_file(file); @@ -4000,14 +4007,14 @@ static int selinux_file_ioctl(struct file *file, unsigned int cmd, case FIOCLEX: case FIONCLEX: if (!selinux_policycap_ioctl_skip_cloexec()) - error = ioctl_has_perm(cred, file, FILE__IOCTL, (u16) cmd); + error = ioctl_has_perm(cred, file, FILE__IOCTL, cmd); break; /* default case assumes that the command will go * to the file's ioctl() function. */ default: - error = ioctl_has_perm(cred, file, FILE__IOCTL, (u16) cmd); + error = ioctl_has_perm(cred, file, FILE__IOCTL, cmd); } return error; } diff --git a/security/selinux/include/avc.h b/security/selinux/include/avc.h index 01b5167fee1a89..87027a934a118d 100644 --- a/security/selinux/include/avc.h +++ b/security/selinux/include/avc.h @@ -61,8 +61,9 @@ struct selinux_audit_data { void __init avc_init(void); -static inline u32 avc_audit_required(u32 requested, struct av_decision *avd, - int result, u32 auditdeny, u32 *deniedp) +static inline u32 avc_audit_required(u32 requested, + const struct av_decision *avd, int result, + u32 auditdeny, u32 *deniedp) { u32 denied, audited; @@ -121,7 +122,7 @@ int slow_avc_audit(u32 ssid, u32 tsid, u16 tclass, u32 requested, u32 audited, * before calling the auditing code. */ static inline int avc_audit(u32 ssid, u32 tsid, u16 tclass, u32 requested, - struct av_decision *avd, int result, + const struct av_decision *avd, int result, struct common_audit_data *a) { u32 audited, denied; diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index c7d91476971cb5..4256838d496c0c 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -75,8 +75,8 @@ struct selinux_fs_info { char **bool_pending_names; int *bool_pending_values; struct dentry *class_dir; - unsigned long last_class_ino; - unsigned long last_ino; + u64 last_class_ino; + u64 last_ino; struct super_block *sb; }; @@ -247,11 +247,6 @@ static int sel_mmap_handle_status(struct file *filp, /* only allows one page from the head */ if (vma->vm_pgoff > 0 || size != PAGE_SIZE) return -EIO; - /* disallow writable mapping */ - if (vma->vm_flags & VM_WRITE) - return -EPERM; - /* disallow mprotect() turns it into writable */ - vm_flags_clear(vma, VM_MAYWRITE); return remap_pfn_range(vma, vma->vm_start, page_to_pfn(status), @@ -304,15 +299,14 @@ static int sel_make_bools(struct selinux_policy *newpolicy, struct dentry *bool_ int **bool_pending_values); static int sel_make_classes(struct selinux_policy *newpolicy, struct dentry *class_dir, - unsigned long *last_class_ino); + u64 *last_class_ino); /* declaration for sel_make_class_dirs */ static struct dentry *sel_make_dir(struct dentry *dir, const char *name, - unsigned long *ino); + u64 *ino); /* declaration for sel_make_policy_nodes */ -static struct dentry *sel_make_swapover_dir(struct super_block *sb, - unsigned long *ino); +static struct dentry *sel_make_swapover_dir(struct super_block *sb, u64 *ino); static ssize_t sel_read_mls(struct file *filp, char __user *buf, size_t count, loff_t *ppos) @@ -467,7 +461,7 @@ static int sel_make_policy_nodes(struct selinux_fs_info *fsi, unsigned int bool_num = 0; char **bool_names = NULL; int *bool_values = NULL; - unsigned long tmp_ino = fsi->last_ino; /* Don't increment last_ino in this function */ + u64 tmp_ino = fsi->last_ino; /* Don't increment last_ino in this function */ tmp_parent = sel_make_swapover_dir(fsi->sb, &tmp_ino); if (IS_ERR(tmp_parent)) @@ -1568,30 +1562,32 @@ static int sel_make_initcon_files(struct dentry *dir) return err; } -static inline unsigned long sel_class_to_ino(u16 class) +static inline u64 sel_class_to_ino(u16 class) { return (class * (SEL_VEC_MAX + 1)) | SEL_CLASS_INO_OFFSET; } -static inline u16 sel_ino_to_class(unsigned long ino) +static inline u16 sel_ino_to_class(u64 ino) { - return (ino & SEL_INO_MASK) / (SEL_VEC_MAX + 1); + u32 ino_masked = ino & SEL_INO_MASK; + return ino_masked / (SEL_VEC_MAX + 1); } -static inline unsigned long sel_perm_to_ino(u16 class, u32 perm) +static inline u64 sel_perm_to_ino(u16 class, u32 perm) { return (class * (SEL_VEC_MAX + 1) + perm) | SEL_CLASS_INO_OFFSET; } -static inline u32 sel_ino_to_perm(unsigned long ino) +static inline u32 sel_ino_to_perm(u64 ino) { - return (ino & SEL_INO_MASK) % (SEL_VEC_MAX + 1); + u32 ino_masked = ino & SEL_INO_MASK; + return ino_masked % (SEL_VEC_MAX + 1); } static ssize_t sel_read_class(struct file *file, char __user *buf, size_t count, loff_t *ppos) { - unsigned long ino = file_inode(file)->i_ino; + u64 ino = file_inode(file)->i_ino; char res[TMPBUFLEN]; ssize_t len = scnprintf(res, sizeof(res), "%d", sel_ino_to_class(ino)); return simple_read_from_buffer(buf, count, ppos, res, len); @@ -1605,7 +1601,7 @@ static const struct file_operations sel_class_ops = { static ssize_t sel_read_perm(struct file *file, char __user *buf, size_t count, loff_t *ppos) { - unsigned long ino = file_inode(file)->i_ino; + u64 ino = file_inode(file)->i_ino; char res[TMPBUFLEN]; ssize_t len = scnprintf(res, sizeof(res), "%d", sel_ino_to_perm(ino)); return simple_read_from_buffer(buf, count, ppos, res, len); @@ -1622,7 +1618,7 @@ static ssize_t sel_read_policycap(struct file *file, char __user *buf, int value; char tmpbuf[TMPBUFLEN]; ssize_t length; - unsigned long i_ino = file_inode(file)->i_ino; + u64 i_ino = file_inode(file)->i_ino; value = security_policycap_supported(i_ino & SEL_INO_MASK); length = scnprintf(tmpbuf, TMPBUFLEN, "%d", value); @@ -1698,7 +1694,7 @@ static int sel_make_class_dir_entries(struct selinux_policy *newpolicy, static int sel_make_classes(struct selinux_policy *newpolicy, struct dentry *class_dir, - unsigned long *last_class_ino) + u64 *last_class_ino) { u32 i, nclasses; int rc; @@ -1763,7 +1759,7 @@ static int sel_make_policycap(struct dentry *dir) } static struct dentry *sel_make_dir(struct dentry *dir, const char *name, - unsigned long *ino) + u64 *ino) { struct inode *inode; @@ -1782,7 +1778,7 @@ static struct dentry *sel_make_dir(struct dentry *dir, const char *name, return sel_attach(dir, name, inode); } -static int reject_all(struct mnt_idmap *idmap, struct inode *inode, int mask) +static int reject_all(const struct mnt_idmap *idmap, struct inode *inode, int mask) { return -EPERM; // no access for anyone, root or no root. } @@ -1792,8 +1788,7 @@ static const struct inode_operations swapover_dir_inode_operations = { .permission = reject_all, }; -static struct dentry *sel_make_swapover_dir(struct super_block *sb, - unsigned long *ino) +static struct dentry *sel_make_swapover_dir(struct super_block *sb, u64 *ino) { struct dentry *dentry; struct inode *inode; @@ -1820,6 +1815,17 @@ static struct dentry *sel_make_swapover_dir(struct super_block *sb, #define NULL_FILE_NAME "null" +static void sel_mark_immutable(struct dentry *root, const char *name) +{ + struct qstr q = QSTR(name); + struct dentry *dentry = try_lookup_noperm(&q, root); + + if (!IS_ERR_OR_NULL(dentry)) { + d_inode(dentry)->i_flags |= S_IMMUTABLE; + dput(dentry); + } +} + static int sel_fill_super(struct super_block *sb, struct fs_context *fc) { struct selinux_fs_info *fsi; @@ -1859,6 +1865,9 @@ static int sel_fill_super(struct super_block *sb, struct fs_context *fc) if (ret) goto err; + sel_mark_immutable(sb->s_root, "status"); + sel_mark_immutable(sb->s_root, "policy"); + fsi = sb->s_fs_info; fsi->bool_dir = sel_make_dir(sb->s_root, BOOL_DIR_NAME, &fsi->last_ino); if (IS_ERR(fsi->bool_dir)) { diff --git a/security/selinux/ss/mls.c b/security/selinux/ss/mls.c index 40c62600650e8e..b48f8256dd4cf9 100644 --- a/security/selinux/ss/mls.c +++ b/security/selinux/ss/mls.c @@ -400,44 +400,6 @@ int mls_range_set(struct context *context, struct mls_range *range) return rc; } -int mls_setup_user_range(struct policydb *p, struct context *fromcon, - struct user_datum *user, struct context *usercon) -{ - if (p->mls_enabled) { - struct mls_level *fromcon_sen = &(fromcon->range.level[0]); - struct mls_level *fromcon_clr = &(fromcon->range.level[1]); - struct mls_level *user_low = &(user->range.level[0]); - struct mls_level *user_clr = &(user->range.level[1]); - struct mls_level *user_def = &(user->dfltlevel); - struct mls_level *usercon_sen = &(usercon->range.level[0]); - struct mls_level *usercon_clr = &(usercon->range.level[1]); - - /* Honor the user's default level if we can */ - if (mls_level_between(user_def, fromcon_sen, fromcon_clr)) - *usercon_sen = *user_def; - else if (mls_level_between(fromcon_sen, user_def, user_clr)) - *usercon_sen = *fromcon_sen; - else if (mls_level_between(fromcon_clr, user_low, user_def)) - *usercon_sen = *user_low; - else - return -EINVAL; - - /* Lower the clearance of available contexts - if the clearance of "fromcon" is lower than - that of the user's default clearance (but - only if the "fromcon" clearance dominates - the user's computed sensitivity level) */ - if (mls_level_dom(user_clr, fromcon_clr)) - *usercon_clr = *fromcon_clr; - else if (mls_level_dom(fromcon_clr, user_clr)) - *usercon_clr = *user_clr; - else - return -EINVAL; - } - - return 0; -} - /* * Convert the MLS fields in the security context * structure `oldc' from the values specified in the diff --git a/security/selinux/ss/mls.h b/security/selinux/ss/mls.h index 93cde1b22992ca..c107b5182fb7b3 100644 --- a/security/selinux/ss/mls.h +++ b/security/selinux/ss/mls.h @@ -46,9 +46,6 @@ int mls_compute_sid(struct policydb *p, struct context *scontext, struct context *tcontext, u16 tclass, u32 specified, struct context *newcontext, bool sock); -int mls_setup_user_range(struct policydb *p, struct context *fromcon, - struct user_datum *user, struct context *usercon); - #ifdef CONFIG_NETLABEL void mls_export_netlbl_lvl(struct policydb *p, struct context *context, struct netlbl_lsm_secattr *secattr); diff --git a/security/selinux/ss/mls_types.h b/security/selinux/ss/mls_types.h index 51df2ebd12115e..d36a03f104c231 100644 --- a/security/selinux/ss/mls_types.h +++ b/security/selinux/ss/mls_types.h @@ -42,9 +42,6 @@ static inline int mls_level_dom(const struct mls_level *l1, #define mls_level_incomp(l1, l2) \ (!mls_level_dom((l1), (l2)) && !mls_level_dom((l2), (l1))) -#define mls_level_between(l1, l2, l3) \ - (mls_level_dom((l1), (l2)) && mls_level_dom((l3), (l1))) - #define mls_range_contains(r1, r2) \ (mls_level_dom(&(r2).level[0], &(r1).level[0]) && \ mls_level_dom(&(r1).level[1], &(r2).level[1])) diff --git a/security/selinux/ss/policydb.c b/security/selinux/ss/policydb.c index b75c977c6673db..fe0c0792583f86 100644 --- a/security/selinux/ss/policydb.c +++ b/security/selinux/ss/policydb.c @@ -560,6 +560,9 @@ static int common_index(void *key, void *datum, void *datap) if (!comdatum->value || comdatum->value > p->p_commons.nprim) return -EINVAL; + if (p->sym_val_to_name[SYM_COMMONS][comdatum->value - 1]) + return -EINVAL; + p->sym_val_to_name[SYM_COMMONS][comdatum->value - 1] = key; return 0; @@ -575,6 +578,9 @@ static int class_index(void *key, void *datum, void *datap) if (!cladatum->value || cladatum->value > p->p_classes.nprim) return -EINVAL; + if (p->class_val_to_struct[cladatum->value - 1]) + return -EINVAL; + p->sym_val_to_name[SYM_CLASSES][cladatum->value - 1] = key; p->class_val_to_struct[cladatum->value - 1] = cladatum; return 0; @@ -591,6 +597,9 @@ static int role_index(void *key, void *datum, void *datap) role->bounds > p->p_roles.nprim) return -EINVAL; + if (p->role_val_to_struct[role->value - 1]) + return -EINVAL; + p->sym_val_to_name[SYM_ROLES][role->value - 1] = key; p->role_val_to_struct[role->value - 1] = role; return 0; @@ -613,6 +622,8 @@ static int type_index(void *key, void *datum, void *datap) } if (typdatum->primary) { + if (p->type_val_to_struct[typdatum->value - 1]) + return -EINVAL; p->sym_val_to_name[SYM_TYPES][typdatum->value - 1] = key; p->type_val_to_struct[typdatum->value - 1] = typdatum; } @@ -631,6 +642,9 @@ static int user_index(void *key, void *datum, void *datap) usrdatum->bounds > p->p_users.nprim) return -EINVAL; + if (p->user_val_to_struct[usrdatum->value - 1]) + return -EINVAL; + p->sym_val_to_name[SYM_USERS][usrdatum->value - 1] = key; p->user_val_to_struct[usrdatum->value - 1] = usrdatum; return 0; @@ -647,8 +661,11 @@ static int sens_index(void *key, void *datum, void *datap) if (!levdatum->level.sens || levdatum->level.sens > p->p_levels.nprim) return -EINVAL; - if (!levdatum->isalias) + if (!levdatum->isalias) { + if (p->sym_val_to_name[SYM_LEVELS][levdatum->level.sens - 1]) + return -EINVAL; p->sym_val_to_name[SYM_LEVELS][levdatum->level.sens - 1] = key; + } return 0; } @@ -664,8 +681,11 @@ static int cat_index(void *key, void *datum, void *datap) if (!catdatum->value || catdatum->value > p->p_cats.nprim) return -EINVAL; - if (!catdatum->isalias) + if (!catdatum->isalias) { + if (p->sym_val_to_name[SYM_CATS][catdatum->value - 1]) + return -EINVAL; p->sym_val_to_name[SYM_CATS][catdatum->value - 1] = key; + } return 0; } @@ -687,6 +707,64 @@ static int sens_cat_index_check(void *key, void *datum, void *datap) return 0; } +static int role_index_check(void *key, void *datum, void *datap) +{ + const struct policydb *p = datap; + const struct role_datum *role = datum; + struct ebitmap_node *node; + u32 bit; + + ebitmap_for_each_positive_bit(&role->dominates, node, bit) { + if (!policydb_role_isvalid(p, bit + 1)) { + pr_err("SELinux: role %s declares dominance on undefined role %u\n", + (const char *)key, bit + 1); + return -EINVAL; + } + } + + ebitmap_for_each_positive_bit(&role->types, node, bit) { + if (!policydb_type_isvalid(p, bit + 1)) { + pr_err("SELinux: role %s authorizes undefined type %u\n", + (const char *)key, bit + 1); + return -EINVAL; + } + } + + return 0; +} + +static int user_index_check(void *key, void *datum, void *datap) +{ + const struct policydb *p = datap; + const struct user_datum *usr = datum; + struct ebitmap_node *node; + u32 bit; + + ebitmap_for_each_positive_bit(&usr->roles, node, bit) { + if (!policydb_role_isvalid(p, bit + 1)) { + pr_err("SELinux: user %s authorizes undefined role %u\n", + (const char *)key, bit + 1); + return -EINVAL; + } + } + + if (p->mls_enabled) { + if (!mls_range_isvalid(p, &usr->range)) { + pr_err("SELinux: user %s has an invalid MLS range\n", + (const char *)key); + return -EINVAL; + } + + if (!mls_level_isvalid(p, &usr->dfltlevel)) { + pr_err("SELinux: user %s has an invalid MLS default level\n", + (const char *)key); + return -EINVAL; + } + } + + return 0; +} + /* clang-format off */ static int (*const index_f[SYM_NUM])(void *key, void *datum, void *datap) = { common_index, @@ -740,8 +818,9 @@ static inline void symtab_hash_eval(struct symtab *s) */ static int policydb_index(struct policydb *p) { + struct ebitmap_node *node; int i, rc; - u32 v; + u32 bit, v; if (p->mls_enabled) pr_debug( @@ -816,6 +895,31 @@ static int policydb_index(struct policydb *p) goto out; } + rc = hashtab_map(&p->p_roles.table, role_index_check, p); + if (rc) + goto out; + + rc = hashtab_map(&p->p_users.table, user_index_check, p); + if (rc) + goto out; + + ebitmap_for_each_positive_bit(&p->permissive_map, node, bit) { + if (!policydb_simpletype_isvalid(p, bit)) { + pr_err("SELinux: permissive map refers to invalid type %u\n", + bit); + rc = -EINVAL; + goto out; + } + } + ebitmap_for_each_positive_bit(&p->neveraudit_map, node, bit) { + if (!policydb_simpletype_isvalid(p, bit)) { + pr_err("SELinux: neveraudit map refers to invalid type %u\n", + bit); + rc = -EINVAL; + goto out; + } + } + rc = 0; out: return rc; @@ -1037,13 +1141,13 @@ bool policydb_context_isvalid(const struct policydb *p, const struct context *c) const struct role_datum *role; const struct user_datum *usrdatum; - if (!c->role || c->role > p->p_roles.nprim) + if (!policydb_role_isvalid(p, c->role)) return false; - if (!c->user || c->user > p->p_users.nprim) + if (!policydb_user_isvalid(p, c->user)) return false; - if (!c->type || c->type > p->p_types.nprim) + if (!policydb_simpletype_isvalid(p, c->type)) return false; if (c->role != OBJECT_R_VAL) { @@ -1051,7 +1155,7 @@ bool policydb_context_isvalid(const struct policydb *p, const struct context *c) * Role must be authorized for the type. */ role = p->role_val_to_struct[c->role - 1]; - if (!role || !ebitmap_get_bit(&role->types, c->type - 1)) + if (!ebitmap_get_bit(&role->types, c->type - 1)) /* role may not be associated with type */ return false; @@ -1059,9 +1163,6 @@ bool policydb_context_isvalid(const struct policydb *p, const struct context *c) * User must be authorized for the role. */ usrdatum = p->user_val_to_struct[c->user - 1]; - if (!usrdatum) - return false; - if (!ebitmap_get_bit(&usrdatum->roles, c->role - 1)) /* user may not be associated with role */ return false; @@ -2350,6 +2451,10 @@ static int filename_trans_read_helper(struct policydb *p, struct policy_file *fp if (rc) goto out; + rc = -EINVAL; + if (ebitmap_get_highest_set_bit(&datum->stypes) >= p->p_types.nprim) + goto out; + rc = next_entry(buf, fp, sizeof(u32)); if (rc) goto out; @@ -2847,7 +2952,8 @@ int policydb_read(struct policydb *p, struct policy_file *fp) goto bad; } - if ((le32_to_cpu(buf[1]) & POLICYDB_CONFIG_MLS)) { + val = le32_to_cpu(buf[1]); + if (val & POLICYDB_CONFIG_MLS) { p->mls_enabled = 1; rc = -EINVAL; @@ -2858,8 +2964,13 @@ int policydb_read(struct policydb *p, struct policy_file *fp) goto bad; } } - p->reject_unknown = !!(le32_to_cpu(buf[1]) & REJECT_UNKNOWN); - p->allow_unknown = !!(le32_to_cpu(buf[1]) & ALLOW_UNKNOWN); + rc = -EINVAL; + if ((val & (REJECT_UNKNOWN | ALLOW_UNKNOWN)) == (REJECT_UNKNOWN | ALLOW_UNKNOWN)) { + pr_err("SELinux: policydb configuration both rejects and allows unknown classes and permissions\n"); + goto bad; + } + p->reject_unknown = !!(val & REJECT_UNKNOWN); + p->allow_unknown = !!(val & ALLOW_UNKNOWN); if (p->policyvers >= POLICYDB_VERSION_POLCAP) { rc = ebitmap_read(&p->policycaps, fp); diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index 8e88ac65fd7fcf..28a8e12603cecc 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -1089,14 +1089,15 @@ static int smack_inode_init_security(struct inode *inode, struct inode *dir, /** * smack_inode_link - Smack check on link + * @idmap: idmap of the mount * @old_dentry: the existing object * @dir: unused * @new_dentry: the new object * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_link(struct dentry *old_dentry, struct inode *dir, - struct dentry *new_dentry) +static int smack_inode_link(const struct mnt_idmap *idmap, struct dentry *old_dentry, + struct inode *dir, struct dentry *new_dentry) { struct smack_known *isp; struct smk_audit_info ad; @@ -1226,6 +1227,7 @@ static int smack_inode_rename(struct inode *old_inode, /** * smack_inode_permission - Smack version of permission() + * @idmap: idmap of the mount * @inode: the inode in question * @mask: the access requested * @@ -1233,7 +1235,8 @@ static int smack_inode_rename(struct inode *old_inode, * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_permission(struct inode *inode, int mask) +static int smack_inode_permission(const struct mnt_idmap *idmap, struct inode *inode, + int mask) { struct superblock_smack *sbsp = smack_superblock(inode->i_sb); struct smk_audit_info ad; @@ -1270,7 +1273,7 @@ static int smack_inode_permission(struct inode *inode, int mask) * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_setattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int smack_inode_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *iattr) { struct smk_audit_info ad; @@ -1348,7 +1351,7 @@ static int smack_inode_xattr_skipcap(const char *name) * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_setxattr(struct mnt_idmap *idmap, +static int smack_inode_setxattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name, const void *value, size_t size, int flags) { @@ -1482,7 +1485,7 @@ static int smack_inode_getxattr(struct dentry *dentry, const char *name) * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_removexattr(struct mnt_idmap *idmap, +static int smack_inode_removexattr(const struct mnt_idmap *idmap, struct dentry *dentry, const char *name) { struct inode_smack *isp; @@ -1543,7 +1546,7 @@ static int smack_inode_removexattr(struct mnt_idmap *idmap, * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_set_acl(struct mnt_idmap *idmap, +static int smack_inode_set_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name, struct posix_acl *kacl) { @@ -1566,7 +1569,7 @@ static int smack_inode_set_acl(struct mnt_idmap *idmap, * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_get_acl(struct mnt_idmap *idmap, +static int smack_inode_get_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { struct smk_audit_info ad; @@ -1588,7 +1591,7 @@ static int smack_inode_get_acl(struct mnt_idmap *idmap, * * Returns 0 if access is permitted, an error code otherwise */ -static int smack_inode_remove_acl(struct mnt_idmap *idmap, +static int smack_inode_remove_acl(const struct mnt_idmap *idmap, struct dentry *dentry, const char *acl_name) { struct smk_audit_info ad; @@ -1612,7 +1615,7 @@ static int smack_inode_remove_acl(struct mnt_idmap *idmap, * * Returns the size of the attribute or an error code */ -static int smack_inode_getsecurity(struct mnt_idmap *idmap, +static int smack_inode_getsecurity(const struct mnt_idmap *idmap, struct inode *inode, const char *name, void **buffer, bool alloc) { diff --git a/sound/hda/codecs/side-codecs/aw88399_hda.c b/sound/hda/codecs/side-codecs/aw88399_hda.c index 11cef49230242d..439ad45557a4d2 100644 --- a/sound/hda/codecs/side-codecs/aw88399_hda.c +++ b/sound/hda/codecs/side-codecs/aw88399_hda.c @@ -209,8 +209,7 @@ static int aw88399_hda_acpi_probe(struct aw88399_hda *aw88399) return -ENODEV; } - struct device *physdev __free(put_device) = - get_device(acpi_get_first_physical_node(adev)); + struct device *physdev __free(put_device) = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!physdev) return -ENODEV; diff --git a/sound/hda/codecs/side-codecs/cs35l41_hda.c b/sound/hda/codecs/side-codecs/cs35l41_hda.c index 62296be45c55e7..32abb71dd40195 100644 --- a/sound/hda/codecs/side-codecs/cs35l41_hda.c +++ b/sound/hda/codecs/side-codecs/cs35l41_hda.c @@ -1926,8 +1926,7 @@ static int cs35l41_hda_read_acpi(struct cs35l41_hda *cs35l41, const char *hid, i } cs35l41->dacpi = adev; - struct device *physdev __free(put_device) = - get_device(acpi_get_first_physical_node(adev)); + struct device *physdev __free(put_device) = acpi_bus_get_primary_device(adev); if (!physdev) { acpi_dev_put(adev); return -ENODEV; diff --git a/sound/hda/codecs/side-codecs/tas2781_hda_i2c.c b/sound/hda/codecs/side-codecs/tas2781_hda_i2c.c index 5e5f46c9fb6b7e..161ca965c63e53 100644 --- a/sound/hda/codecs/side-codecs/tas2781_hda_i2c.c +++ b/sound/hda/codecs/side-codecs/tas2781_hda_i2c.c @@ -100,8 +100,7 @@ static int tas2781_read_acpi(struct tasdevice_priv *p, const char *hid) return -ENODEV; } - struct device *physdev __free(put_device) = - get_device(acpi_get_first_physical_node(adev)); + struct device *physdev __free(put_device) = acpi_bus_get_primary_device(adev); ret = acpi_dev_get_resources(adev, &resources, tas2781_get_i2c_res, p); if (ret < 0) { dev_err(p->dev, "Failed to get ACPI resource.\n"); diff --git a/sound/hda/codecs/side-codecs/tas2781_hda_spi.c b/sound/hda/codecs/side-codecs/tas2781_hda_spi.c index da71b57a26ee0d..f3ba7132d2b874 100644 --- a/sound/hda/codecs/side-codecs/tas2781_hda_spi.c +++ b/sound/hda/codecs/side-codecs/tas2781_hda_spi.c @@ -341,8 +341,7 @@ static int tas2781_read_acpi(struct tas2781_hda *tas_hda, strscpy(p->dev_name, hid, sizeof(p->dev_name)); - struct device *physdev __free(put_device) = - get_device(acpi_get_first_physical_node(adev)); + struct device *physdev __free(put_device) = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!physdev) return -ENODEV; diff --git a/sound/soc/amd/acp-es8336.c b/sound/soc/amd/acp-es8336.c index 9f3f11256788e2..34e953d1a3a76c 100644 --- a/sound/soc/amd/acp-es8336.c +++ b/sound/soc/amd/acp-es8336.c @@ -198,7 +198,7 @@ static int st_es8336_late_probe(struct snd_soc_card *card) if (!adev) return -ENODEV; - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!codec_dev) { dev_err(card->dev, "can not find codec dev\n"); diff --git a/sound/soc/amd/acp/acp3x-es83xx/acp3x-es83xx.c b/sound/soc/amd/acp/acp3x-es83xx/acp3x-es83xx.c index 3a640e65231420..c522b421e1ec2f 100644 --- a/sound/soc/amd/acp/acp3x-es83xx/acp3x-es83xx.c +++ b/sound/soc/amd/acp/acp3x-es83xx/acp3x-es83xx.c @@ -428,7 +428,7 @@ static int acp3x_es83xx_probe(struct snd_soc_card *card) return -ENXIO; } - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!codec_dev) { dev_warn(dev, "Error cannot find codec device, will defer probe\n"); diff --git a/sound/soc/intel/boards/bytcht_es8316.c b/sound/soc/intel/boards/bytcht_es8316.c index ea387dc7427382..4d8d1d176a49f3 100644 --- a/sound/soc/intel/boards/bytcht_es8316.c +++ b/sound/soc/intel/boards/bytcht_es8316.c @@ -601,11 +601,11 @@ static int snd_byt_cht_es8316_mc_probe(struct platform_device *pdev) return -ENOENT; } - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!codec_dev) return -EPROBE_DEFER; - priv->codec_dev = get_device(codec_dev); + priv->codec_dev = codec_dev; /* override platform name, if required */ byt_cht_es8316_card.dev = dev; diff --git a/sound/soc/intel/boards/bytcr_rt5640.c b/sound/soc/intel/boards/bytcr_rt5640.c index 40da3eea5fa72e..c84b9a0fe65a0a 100644 --- a/sound/soc/intel/boards/bytcr_rt5640.c +++ b/sound/soc/intel/boards/bytcr_rt5640.c @@ -1744,11 +1744,11 @@ static int snd_byt_rt5640_mc_probe(struct platform_device *pdev) return -ENOENT; } - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (codec_dev) { - priv->codec_dev = get_device(codec_dev); + priv->codec_dev = codec_dev; } else { /* * Special case for Android tablets where the codec i2c_client diff --git a/sound/soc/intel/boards/bytcr_rt5651.c b/sound/soc/intel/boards/bytcr_rt5651.c index c9b1053205de55..fbfaf0b351c9f6 100644 --- a/sound/soc/intel/boards/bytcr_rt5651.c +++ b/sound/soc/intel/boards/bytcr_rt5651.c @@ -960,11 +960,11 @@ static int snd_byt_rt5651_mc_probe(struct platform_device *pdev) return -ENOENT; } - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!codec_dev) return -EPROBE_DEFER; - priv->codec_dev = get_device(codec_dev); + priv->codec_dev = codec_dev; /* * swap SSP0 if bytcr is detected diff --git a/sound/soc/intel/boards/cht_bsw_rt5645.c b/sound/soc/intel/boards/cht_bsw_rt5645.c index 249be121be1574..4f20928a3604a8 100644 --- a/sound/soc/intel/boards/cht_bsw_rt5645.c +++ b/sound/soc/intel/boards/cht_bsw_rt5645.c @@ -529,7 +529,6 @@ static int snd_cht_mc_probe(struct platform_device *pdev) const char *platform_name; struct cht_mc_private *drv; struct acpi_device *adev; - struct device *codec_dev; bool sof_parent; bool found = false; bool is_bytcr = false; @@ -583,8 +582,8 @@ static int snd_cht_mc_probe(struct platform_device *pdev) return -ENOENT; } - /* acpi_get_first_physical_node() returns a borrowed ref, no need to deref */ - codec_dev = acpi_get_first_physical_node(adev); + struct device *codec_dev __free(put_device) = acpi_bus_get_primary_device(adev); + acpi_dev_put(adev); if (!codec_dev) return -EPROBE_DEFER; diff --git a/sound/soc/intel/boards/sof_cirrus_common.c b/sound/soc/intel/boards/sof_cirrus_common.c index 88fc6cb2bfd49b..a4ecb1f9a61549 100644 --- a/sound/soc/intel/boards/sof_cirrus_common.c +++ b/sound/soc/intel/boards/sof_cirrus_common.c @@ -168,7 +168,7 @@ static int cs35l41_compute_codec_conf(void) cs35l41_name_prefixes[uid]); continue; } - physdev = get_device(acpi_get_first_physical_node(adev)); + physdev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!physdev) { pr_devel("Cannot find physical node for HID %s UID %u (%s)\n", CS35L41_HID, diff --git a/sound/soc/intel/boards/sof_es8336.c b/sound/soc/intel/boards/sof_es8336.c index f1e62c2e79feff..d40f10c6ae54a7 100644 --- a/sound/soc/intel/boards/sof_es8336.c +++ b/sound/soc/intel/boards/sof_es8336.c @@ -702,11 +702,11 @@ static int sof_es8336_probe(struct platform_device *pdev) return -ENOENT; } - codec_dev = acpi_get_first_physical_node(adev); + codec_dev = acpi_bus_get_primary_device(adev); acpi_dev_put(adev); if (!codec_dev) return -EPROBE_DEFER; - priv->codec_dev = get_device(codec_dev); + priv->codec_dev = codec_dev; ret = snd_soc_fixup_dai_links_platform_name(&sof_es8336_card, mach->mach_params.platform); diff --git a/sound/soc/loongson/loongson_card.c b/sound/soc/loongson/loongson_card.c index 6422cc1703b6ad..9f390727846298 100644 --- a/sound/soc/loongson/loongson_card.c +++ b/sound/soc/loongson/loongson_card.c @@ -196,7 +196,6 @@ static int loongson_card_parse_acpi(struct loongson_card_data *data) struct snd_soc_card *card = &data->snd_card; const char *codec_dai_name; struct acpi_device *adev; - struct device *phy_dev; int i, ret; /* fixup platform name based on reference node */ @@ -204,7 +203,7 @@ static int loongson_card_parse_acpi(struct loongson_card_data *data) if (!adev) return -ENOENT; - phy_dev = acpi_get_first_physical_node(adev); + struct device *phy_dev __free(put_device) = acpi_bus_get_primary_device(adev); if (!phy_dev) return -EPROBE_DEFER; diff --git a/tools/include/linux/bitfield-fix-width.h b/tools/include/linux/bitfield-fix-width.h new file mode 100644 index 00000000000000..8c2a0fa224e997 --- /dev/null +++ b/tools/include/linux/bitfield-fix-width.h @@ -0,0 +1,236 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _LINUX_BITFIELD_FIX_WIDTH_H +#define _LINUX_BITFIELD_FIX_WIDTH_H + +#include +#include +#include +#include + +extern void __compiletime_error("value doesn't fit into mask") +__field_overflow(void); + +extern void __compiletime_error("bad bitfield mask") +__bad_mask(void); + +static __always_inline +u64 field_multiplier(u64 field) +{ + if ((field | (field - 1)) & ((field | (field - 1)) + 1)) + __bad_mask(); + return field & -field; +} + +static __always_inline +u64 field_mask(u64 field) +{ + return field / field_multiplier(field); +} + +#define field_max(field) ((typeof(field))field_mask(field)) + +#define __assert_field(v, field) \ + do { \ + if (__builtin_constant_p(v) && \ + ((v) & ~field_mask(field))) \ + __field_overflow(); \ + } while (0) + +static __always_inline +__u8 u8_encode_bits(u8 v, u8 field) +{ + __assert_field(v, field); + return (v & field_mask(field)) * field_multiplier(field); +} + +static __always_inline +__u8 u8_replace_bits(__u8 old, u8 val, u8 field) +{ + return (old & ~field) | u8_encode_bits(val, field); +} + +static __always_inline +void u8p_replace_bits(__u8 *p, u8 val, u8 field) +{ + *p = (*p & ~field) | u8_encode_bits(val, field); +} + +static __always_inline +u8 u8_get_bits(__u8 v, u8 field) +{ + return (v & field) / field_multiplier(field); +} + +static __always_inline +__le16 le16_encode_bits(u16 v, u16 field) +{ + __assert_field(v, field); + return cpu_to_le16((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +void le16p_replace_bits(__le16 *p, u16 val, u16 field) +{ + *p = (*p & ~cpu_to_le16(field)) | le16_encode_bits(val, field); +} + +static __always_inline +u16 le16_get_bits(__le16 v, u16 field) +{ + return (le16_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__be16 be16_encode_bits(u16 v, u16 field) +{ + __assert_field(v, field); + return cpu_to_be16((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +u16 be16_get_bits(__be16 v, u16 field) +{ + return (be16_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__u16 u16_encode_bits(u16 v, u16 field) +{ + __assert_field(v, field); + return (v & field_mask(field)) * field_multiplier(field); +} + +static __always_inline +__u16 u16_replace_bits(__u16 old, u16 val, u16 field) +{ + return (old & ~field) | u16_encode_bits(val, field); +} + +static __always_inline +void u16p_replace_bits(__u16 *p, u16 val, u16 field) +{ + *p = (*p & ~field) | u16_encode_bits(val, field); +} + +static __always_inline +u16 u16_get_bits(__u16 v, u16 field) +{ + return (v & field) / field_multiplier(field); +} + +static __always_inline +__le32 le32_encode_bits(u32 v, u32 field) +{ + __assert_field(v, field); + return cpu_to_le32((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +void le32p_replace_bits(__le32 *p, u32 val, u32 field) +{ + *p = (*p & ~cpu_to_le32(field)) | le32_encode_bits(val, field); +} + +static __always_inline +u32 le32_get_bits(__le32 v, u32 field) +{ + return (le32_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__be32 be32_encode_bits(u32 v, u32 field) +{ + __assert_field(v, field); + return cpu_to_be32((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +void be32p_replace_bits(__be32 *p, u32 val, u32 field) +{ + *p = (*p & ~cpu_to_be32(field)) | be32_encode_bits(val, field); +} + +static __always_inline +u32 be32_get_bits(__be32 v, u32 field) +{ + return (be32_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__u32 u32_encode_bits(u32 v, u32 field) +{ + __assert_field(v, field); + return (v & field_mask(field)) * field_multiplier(field); +} + +static __always_inline +__u32 u32_replace_bits(__u32 old, u32 val, u32 field) +{ + return (old & ~field) | u32_encode_bits(val, field); +} + +static __always_inline +void u32p_replace_bits(__u32 *p, u32 val, u32 field) +{ + *p = (*p & ~field) | u32_encode_bits(val, field); +} + +static __always_inline +u32 u32_get_bits(__u32 v, u32 field) +{ + return (v & field) / field_multiplier(field); +} + +static __always_inline +__le64 le64_encode_bits(u64 v, u64 field) +{ + __assert_field(v, field); + return cpu_to_le64((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +u64 le64_get_bits(__le64 v, u64 field) +{ + return (le64_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__be64 be64_encode_bits(u64 v, u64 field) +{ + __assert_field(v, field); + return cpu_to_be64((v & field_mask(field)) * field_multiplier(field)); +} + +static __always_inline +u64 be64_get_bits(__be64 v, u64 field) +{ + return (be64_to_cpu(v) & field) / field_multiplier(field); +} + +static __always_inline +__u64 u64_encode_bits(u64 v, u64 field) +{ + __assert_field(v, field); + return (v & field_mask(field)) * field_multiplier(field); +} + +static __always_inline +__u64 u64_replace_bits(__u64 old, u64 val, u64 field) +{ + return (old & ~field) | u64_encode_bits(val, field); +} + +static __always_inline +void u64p_replace_bits(__u64 *p, u64 val, u64 field) +{ + *p = (*p & ~field) | u64_encode_bits(val, field); +} + +static __always_inline +u64 u64_get_bits(__u64 v, u64 field) +{ + return (v & field) / field_multiplier(field); +} + +#endif /* _LINUX_BITFIELD_FIX_WIDTH_H */ diff --git a/tools/include/linux/bitfield.h b/tools/include/linux/bitfield.h index ddf81f24956ba0..4508cefabec92d 100644 --- a/tools/include/linux/bitfield.h +++ b/tools/include/linux/bitfield.h @@ -9,7 +9,7 @@ #include #include -#include +#include /* * Bitfield access macros @@ -127,51 +127,4 @@ (typeof(_mask))(((_reg) & (_mask)) >> __bf_shf(_mask)); \ }) -extern void __compiletime_error("value doesn't fit into mask") -__field_overflow(void); -extern void __compiletime_error("bad bitfield mask") -__bad_mask(void); -static __always_inline u64 field_multiplier(u64 field) -{ - if ((field | (field - 1)) & ((field | (field - 1)) + 1)) - __bad_mask(); - return field & -field; -} -static __always_inline u64 field_mask(u64 field) -{ - return field / field_multiplier(field); -} -#define field_max(field) ((typeof(field))field_mask(field)) -#define ____MAKE_OP(type,base,to,from) \ -static __always_inline __##type type##_encode_bits(base v, base field) \ -{ \ - if (__builtin_constant_p(v) && (v & ~field_mask(field))) \ - __field_overflow(); \ - return to((v & field_mask(field)) * field_multiplier(field)); \ -} \ -static __always_inline __##type type##_replace_bits(__##type old, \ - base val, base field) \ -{ \ - return (old & ~to(field)) | type##_encode_bits(val, field); \ -} \ -static __always_inline void type##p_replace_bits(__##type *p, \ - base val, base field) \ -{ \ - *p = (*p & ~to(field)) | type##_encode_bits(val, field); \ -} \ -static __always_inline base type##_get_bits(__##type v, base field) \ -{ \ - return (from(v) & field)/field_multiplier(field); \ -} -#define __MAKE_OP(size) \ - ____MAKE_OP(le##size,u##size,cpu_to_le##size,le##size##_to_cpu) \ - ____MAKE_OP(be##size,u##size,cpu_to_be##size,be##size##_to_cpu) \ - ____MAKE_OP(u##size,u##size,,) -____MAKE_OP(u8,u8,,) -__MAKE_OP(16) -__MAKE_OP(32) -__MAKE_OP(64) -#undef __MAKE_OP -#undef ____MAKE_OP - #endif diff --git a/tools/include/nolibc/Makefile b/tools/include/nolibc/Makefile index 6d213d372bf835..0880be0f6ef587 100644 --- a/tools/include/nolibc/Makefile +++ b/tools/include/nolibc/Makefile @@ -17,7 +17,23 @@ endif # it defaults to this nolibc directory. OUTPUT ?= $(CURDIR)/ -architectures := alpha arm arm64 loongarch m68k mips openrisc parisc powerpc riscv s390 sh sparc x86 +architectures := \ + alpha \ + arm \ + arm64 \ + hexagon \ + loongarch \ + m68k \ + mips \ + openrisc \ + parisc \ + powerpc \ + riscv \ + s390 \ + sh \ + sparc \ + x86 \ + arch_files := arch.h $(addsuffix .h, $(addprefix arch-, $(architectures))) all_files := \ alloca.h \ @@ -59,6 +75,7 @@ all_files := \ sys/reboot.h \ sys/resource.h \ sys/select.h \ + sys/sendfile.h \ sys/stat.h \ sys/syscall.h \ sys/sysmacros.h \ diff --git a/tools/include/nolibc/arch-arm.h b/tools/include/nolibc/arch-arm.h index b88686fb19c435..b5b3456c75cb02 100644 --- a/tools/include/nolibc/arch-arm.h +++ b/tools/include/nolibc/arch-arm.h @@ -196,11 +196,6 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote } #endif /* NOLIBC_NO_RUNTIME */ -static __attribute__((unused)) -int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1) -{ - return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1); -} -#define _sys_ftruncate64 _sys_ftruncate64 +#define __NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR #endif /* _NOLIBC_ARCH_ARM_H */ diff --git a/tools/include/nolibc/arch-hexagon.h b/tools/include/nolibc/arch-hexagon.h new file mode 100644 index 00000000000000..68f73529a6ddb7 --- /dev/null +++ b/tools/include/nolibc/arch-hexagon.h @@ -0,0 +1,159 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * hexagon specific definitions for NOLIBC + * Copyright (C) 2026 Thomas Weißschuh + */ + +#ifndef _NOLIBC_ARCH_HEXAGON_H +#define _NOLIBC_ARCH_HEXAGON_H + +#include + +#include "compiler.h" +#include "crt.h" + +/* + * Syscalls for hexagon: + * - syscall number is passed in r6 + * - arguments are in r0, r1, r2, r3, r4, r5 + * - the system call is performed by calling trap0(#1) + * - syscall return value is in r0 + */ + +#define _NOLIBC_SYSCALL_CLOBBERLIST "memory" + +#define __nolibc_syscall0(num) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0"); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "=r"(_arg1) \ + : "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall1(num, arg1) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall2(num, arg1, arg2) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_arg2), \ + "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall3(num, arg1, arg2, arg3) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + register long _arg3 __asm__ ("r2") = (long)(arg3); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_arg2), "r"(_arg3), \ + "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall4(num, arg1, arg2, arg3, arg4) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + register long _arg3 __asm__ ("r2") = (long)(arg3); \ + register long _arg4 __asm__ ("r3") = (long)(arg4); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_arg2), "r"(_arg3), "r"(_arg4), \ + "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + register long _arg3 __asm__ ("r2") = (long)(arg3); \ + register long _arg4 __asm__ ("r3") = (long)(arg4); \ + register long _arg5 __asm__ ("r4") = (long)(arg5); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \ + "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \ +({ \ + register long _num __asm__ ("r6") = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + register long _arg3 __asm__ ("r2") = (long)(arg3); \ + register long _arg4 __asm__ ("r3") = (long)(arg4); \ + register long _arg5 __asm__ ("r4") = (long)(arg5); \ + register long _arg6 __asm__ ("r5") = (long)(arg6); \ + \ + __asm__ volatile ( \ + "trap0(#1)\n" \ + : "+r"(_arg1) \ + : "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), "r"(_arg6), \ + "r"(_num) \ + : _NOLIBC_SYSCALL_CLOBBERLIST \ + ); \ + _arg1; \ +}) + +#ifndef NOLIBC_NO_RUNTIME +/* startup code */ +void __attribute__((weak, noreturn)) +__nolibc_entrypoint __nolibc_no_stack_protector +_start(void) +{ + __asm__ volatile ( + "r0 = sp\n" /* save stack pointer to r0, as arg1 of _start_c */ + "call _start_c\n" /* transfer to c runtime */ + ); + __nolibc_entrypoint_epilogue(); +} +#endif /* NOLIBC_NO_RUNTIME */ + +#define __NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR + +#endif /* _NOLIBC_ARCH_HEXAGON_H */ diff --git a/tools/include/nolibc/arch-mips.h b/tools/include/nolibc/arch-mips.h index 26ad413cec62b7..c0da0cc309ce14 100644 --- a/tools/include/nolibc/arch-mips.h +++ b/tools/include/nolibc/arch-mips.h @@ -72,6 +72,7 @@ #define _NOLIBC_SYSCALL_STACK_UNRESERVE #define _NOLIBC_SYSCALL_REG register long long +#define __NOLIBC_BITS_PER_SYSCALL_ARG 64 #endif /* _ABIO32 */ @@ -286,12 +287,7 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote #endif /* NOLIBC_NO_RUNTIME */ #if defined(_ABIO32) -static __attribute__((unused)) -int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1) -{ - return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1); -} -#define _sys_ftruncate64 _sys_ftruncate64 +#define __NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR #endif #endif /* _NOLIBC_ARCH_MIPS_H */ diff --git a/tools/include/nolibc/arch-parisc.h b/tools/include/nolibc/arch-parisc.h index 417043ecbc539f..bf57ba16055ca7 100644 --- a/tools/include/nolibc/arch-parisc.h +++ b/tools/include/nolibc/arch-parisc.h @@ -7,9 +7,7 @@ #ifndef _NOLIBC_ARCH_PARISC_H #define _NOLIBC_ARCH_PARISC_H -#if defined(__LP64__) -#error 64-bit not supported -#endif +#include #include "compiler.h" #include "crt.h" @@ -167,9 +165,15 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void) { __asm__ volatile ( +#ifdef __LP64__ + ".import __gp\n" /* Set up the dp register */ + "ldil L%__gp, %dp\n" + "ldo R%__gp(%dp), %dp\n" +#else ".import $global$\n" /* Set up the dp register */ "ldil L%$global$, %dp\n" - "ldo R%$global$(%r27), %dp\n" + "ldo R%$global$(%dp), %dp\n" +#endif "b _start_c\n" /* Call _start_c, the load below is executed first */ @@ -182,4 +186,20 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote } #endif /* NOLIBC_NO_RUNTIME */ +#ifdef __LP64__ +/* + * This override is normally only meant for 32-bit architectures. + * But parisc 64-bit has __NR_ftruncate64, so this path is taken. + * Add an unused second argument to satisfy the calling convention. + */ +static __attribute__((unused)) +int _sys_ftruncate64(int fd, uint64_t length, uint64_t unused) +{ + (void)unused; + + return __nolibc_syscall2(__NR_ftruncate64, fd, length); +} +#define _sys_ftruncate64 _sys_ftruncate64 +#endif + #endif /* _NOLIBC_ARCH_PARISC_H */ diff --git a/tools/include/nolibc/arch-powerpc.h b/tools/include/nolibc/arch-powerpc.h index dbe2e5205aaafa..a4711638cb66fa 100644 --- a/tools/include/nolibc/arch-powerpc.h +++ b/tools/include/nolibc/arch-powerpc.h @@ -222,12 +222,7 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote #endif /* NOLIBC_NO_RUNTIME */ #if !defined(__powerpc64__) -static __attribute__((unused)) -int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1) -{ - return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1); -} -#define _sys_ftruncate64 _sys_ftruncate64 +#define __NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR #endif #endif /* _NOLIBC_ARCH_POWERPC_H */ diff --git a/tools/include/nolibc/arch-sh.h b/tools/include/nolibc/arch-sh.h index a32378fd621fda..996cfbe1bee327 100644 --- a/tools/include/nolibc/arch-sh.h +++ b/tools/include/nolibc/arch-sh.h @@ -10,6 +10,8 @@ #include "compiler.h" #include "crt.h" +#include + /* * Syscalls for SuperH: * - registers are 32bit wide @@ -161,4 +163,20 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote } #endif /* NOLIBC_NO_RUNTIME */ +static __attribute__((unused)) +ssize_t _sys_pread(int fd, void *buf, size_t count, off_t offset) +{ + return __nolibc_syscall6(__NR_pread64, fd, buf, count, 0, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +} +#define _sys_pread _sys_pread + +static __attribute__((unused)) +ssize_t _sys_pwrite(int fd, const void *buf, size_t count, off_t offset) +{ + return __nolibc_syscall6(__NR_pwrite64, fd, buf, count, 0, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +} +#define _sys_pwrite _sys_pwrite + #endif /* _NOLIBC_ARCH_SH_H */ diff --git a/tools/include/nolibc/arch-x86.h b/tools/include/nolibc/arch-x86.h index 577e9b682eb37e..0638f931674936 100644 --- a/tools/include/nolibc/arch-x86.h +++ b/tools/include/nolibc/arch-x86.h @@ -386,5 +386,7 @@ __asm__ ( ".popsection\n" ); +#define __NOLIBC_BITS_PER_SYSCALL_ARG 64 + #endif /* !defined(__x86_64__) */ #endif /* _NOLIBC_ARCH_X86_H */ diff --git a/tools/include/nolibc/arch.h b/tools/include/nolibc/arch.h index 06cc2dbe7a3305..79ed5900f2a79f 100644 --- a/tools/include/nolibc/arch.h +++ b/tools/include/nolibc/arch.h @@ -34,8 +34,15 @@ #include "arch-parisc.h" #elif defined(__alpha__) #include "arch-alpha.h" +#elif defined(__hexagon__) +#include "arch-hexagon.h" #else #error Unsupported Architecture #endif + +#ifndef __NOLIBC_BITS_PER_SYSCALL_ARG +#define __NOLIBC_BITS_PER_SYSCALL_ARG __BITS_PER_LONG +#endif + #endif /* _NOLIBC_ARCH_H */ diff --git a/tools/include/nolibc/compiler.h b/tools/include/nolibc/compiler.h index f2d7a81d0d7c6a..375c45ce81c836 100644 --- a/tools/include/nolibc/compiler.h +++ b/tools/include/nolibc/compiler.h @@ -90,4 +90,13 @@ # define __nolibc_no_sanitize_undefined #endif +/* + * Helper for 32-bit machines where a 64-bit syscall arg needs to be split into + * two 32-bit parts while making sure the order of the low/high parts are correct + * for the endianness: + * __NOLIBC_LLARGPART(x, 0), __NOLIBC_LLARGPART(x, 1) + */ +#define __NOLIBC_LLARGPART(_arg, _part) \ + (((union { long long ll; long l[2]; }) { .ll = _arg }).l[_part]) + #endif /* _NOLIBC_COMPILER_H */ diff --git a/tools/include/nolibc/dirent.h b/tools/include/nolibc/dirent.h index 4e02ef25e72d84..19857369de2a6f 100644 --- a/tools/include/nolibc/dirent.h +++ b/tools/include/nolibc/dirent.h @@ -30,10 +30,23 @@ typedef struct { static __attribute__((unused)) DIR *fdopendir(int fd) { + struct stat buf; + int ret; + if (fd < 0) { SET_ERRNO(EBADF); return NULL; } + + ret = fstat(fd, &buf); + if (ret < 0) + return NULL; + + if (!S_ISDIR(buf.st_mode)) { + SET_ERRNO(ENOTDIR); + return NULL; + } + return (DIR *)(intptr_t)~fd; } @@ -42,10 +55,11 @@ DIR *opendir(const char *name) { int fd; - fd = open(name, O_RDONLY); + fd = open(name, O_RDONLY | O_DIRECTORY); if (fd == -1) return NULL; - return fdopendir(fd); + + return (DIR *)(intptr_t)~fd; } static __attribute__((unused)) @@ -67,6 +81,7 @@ int readdir_r(DIR *dirp, struct dirent *entry, struct dirent **result) struct linux_dirent64 *ldir = (void *)buf; intptr_t i = (intptr_t)dirp; int fd, ret; + off_t off; if (i >= 0) return EBADF; @@ -86,9 +101,9 @@ int readdir_r(DIR *dirp, struct dirent *entry, struct dirent **result) * readdir() can only return one entry at a time. * Make sure the non-returned ones are not skipped. */ - ret = _sys_lseek(fd, ldir->d_off, SEEK_SET); - if (ret < 0) - return -ret; + off = _sys_lseek(fd, ldir->d_off, SEEK_SET); + if (off < 0) + return -off; entry->d_ino = ldir->d_ino; /* the destination should always be big enough */ diff --git a/tools/include/nolibc/err.h b/tools/include/nolibc/err.h index e22ae87a728920..61978d537ed8f8 100644 --- a/tools/include/nolibc/err.h +++ b/tools/include/nolibc/err.h @@ -60,7 +60,7 @@ void verr(int eval, const char *fmt, va_list args) static __attribute__((noreturn, unused)) void verrx(int eval, const char *fmt, va_list args) { - warnx(fmt, args); + vwarnx(fmt, args); exit(eval); } diff --git a/tools/include/nolibc/nolibc.h b/tools/include/nolibc/nolibc.h index faa94f24728165..352a207bbedc65 100644 --- a/tools/include/nolibc/nolibc.h +++ b/tools/include/nolibc/nolibc.h @@ -106,6 +106,7 @@ #include "sys/reboot.h" #include "sys/resource.h" #include "sys/select.h" +#include "sys/sendfile.h" #include "sys/stat.h" #include "sys/syscall.h" #include "sys/sysmacros.h" diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h index 1816c2368b6803..1c5a914c0f2317 100644 --- a/tools/include/nolibc/stdlib.h +++ b/tools/include/nolibc/stdlib.h @@ -130,9 +130,11 @@ void *malloc(size_t len) { struct nolibc_heap *heap; - /* Always allocate memory with size multiple of 4096. */ - len = sizeof(*heap) + len; - len = (len + 4095UL) & -4096UL; + if (__builtin_expect(__builtin_add_overflow(len, sizeof(*heap), &len), 0)) { + SET_ERRNO(ENOMEM); + return NULL; + } + heap = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0); if (__builtin_expect(heap == MAP_FAILED, 0)) @@ -143,7 +145,7 @@ void *malloc(size_t len) } static __attribute__((unused)) -void *calloc(size_t size, size_t nmemb) +void *calloc(size_t nmemb, size_t size) { size_t x; @@ -230,7 +232,7 @@ int _nolibc_u64toa_base(uint64_t in, char *buffer, unsigned int base, uint64_t r /* Generate least significant digit first */ do { -#if defined(__SIZEOF_INT128__) && !defined(__mips__) && !defined(__sparc__) +#if defined(__SIZEOF_INT128__) && !defined(__mips__) && !defined(__sparc__) && !defined(__hppa__) q = ((unsigned __int128)in * recip) >> 64; #else uint64_t p = (uint32_t)in * (recip >> 32); diff --git a/tools/include/nolibc/sys.h b/tools/include/nolibc/sys.h index 548f94d96ed203..b63fa6c618c570 100644 --- a/tools/include/nolibc/sys.h +++ b/tools/include/nolibc/sys.h @@ -71,16 +71,6 @@ static __inline__ int __nolibc_enosys(const char *syscall, ...) #endif -/* - * Helper for 32-bit machines where a 64-bit syscall arg needs to be split into - * two 32-bit parts while making sure the order of the low/high parts are correct - * for the endianness: - * __NOLIBC_LLARGPART(x, 0), __NOLIBC_LLARGPART(x, 1) - */ -#define __NOLIBC_LLARGPART(_arg, _part) \ - (((union { long long ll; long l[2]; }) { .ll = _arg }).l[_part]) - - /* Functions in this file only describe syscalls. They're declared static so * that the compiler usually decides to inline them while still being allowed * to pass a pointer to one of their instances. Each syscall exists in two @@ -755,6 +745,56 @@ int pivot_root(const char *new, const char *old) } +/* + * ssize_t pread(int fd, void *buf, size_t count, off_t offset); + * ssize_t pwrite(int fd, const void *buf, size_t count, off_t offset); + */ + +#ifndef _sys_pread +static __attribute__((unused)) +ssize_t _sys_pread(int fd, void *buf, size_t count, off_t offset) +{ +#if __NOLIBC_BITS_PER_SYSCALL_ARG == 64 + return __nolibc_syscall4(__NR_pread64, fd, buf, count, offset); +#elif defined(__NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR) + return __nolibc_syscall6(__NR_pread64, fd, buf, count, 0, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +#else + return __nolibc_syscall5(__NR_pread64, fd, buf, count, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +#endif +} +#endif /* _sys_pread */ + +static __attribute__((unused)) +ssize_t pread(int fd, void *buf, size_t count, off_t offset) +{ + return __sysret(_sys_pread(fd, buf, count, offset)); +} + +#ifndef _sys_pwrite +static __attribute__((unused)) +ssize_t _sys_pwrite(int fd, const void *buf, size_t count, off_t offset) +{ +#if __NOLIBC_BITS_PER_SYSCALL_ARG == 64 + return __nolibc_syscall4(__NR_pwrite64, fd, buf, count, offset); +#elif defined(__NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR) + return __nolibc_syscall6(__NR_pwrite64, fd, buf, count, 0, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +#else + return __nolibc_syscall5(__NR_pwrite64, fd, buf, count, + __NOLIBC_LLARGPART(offset, 0), __NOLIBC_LLARGPART(offset, 1)); +#endif +} +#endif /* _sys_pwrite */ + +static __attribute__((unused)) +ssize_t pwrite(int fd, const void *buf, size_t count, off_t offset) +{ + return __sysret(_sys_pwrite(fd, buf, count, offset)); +} + + /* * ssize_t read(int fd, void *buf, size_t count); */ diff --git a/tools/include/nolibc/sys/select.h b/tools/include/nolibc/sys/select.h index 6d65d9ef3d6af4..f299c79939f593 100644 --- a/tools/include/nolibc/sys/select.h +++ b/tools/include/nolibc/sys/select.h @@ -26,7 +26,7 @@ typedef struct { int __fd = (fd); \ if (__fd >= 0) \ __set->fds[__fd / FD_SETIDXMASK] &= \ - ~(1U << (__fd & FD_SETBITMASK)); \ + ~(1UL << (__fd & FD_SETBITMASK)); \ } while (0) #define FD_SET(fd, set) do { \ @@ -34,7 +34,7 @@ typedef struct { int __fd = (fd); \ if (__fd >= 0) \ __set->fds[__fd / FD_SETIDXMASK] |= \ - 1 << (__fd & FD_SETBITMASK); \ + 1UL << (__fd & FD_SETBITMASK); \ } while (0) #define FD_ISSET(fd, set) ({ \ @@ -43,7 +43,7 @@ typedef struct { int __r = 0; \ if (__fd >= 0) \ __r = !!(__set->fds[__fd / FD_SETIDXMASK] & \ -1U << (__fd & FD_SETBITMASK)); \ +1UL << (__fd & FD_SETBITMASK)); \ __r; \ }) diff --git a/tools/include/nolibc/sys/sendfile.h b/tools/include/nolibc/sys/sendfile.h new file mode 100644 index 00000000000000..ac7d8584771847 --- /dev/null +++ b/tools/include/nolibc/sys/sendfile.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * sendfile for NOLIBC + * Copyright (C) 2026 Daniel Palmer + */ + +/* make sure to include all global symbols */ +#include "../nolibc.h" + +#ifndef _NOLIBC_SYS_SENDFILE_H +#define _NOLIBC_SYS_SENDFILE_H + +#include "../sys.h" +#include + +/* + * ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count); + */ + +static __attribute__((unused)) +ssize_t _sys_sendfile(int out_fd, int in_fd, off_t *offset, size_t count) +{ + __nolibc_static_assert(sizeof(*offset) == sizeof(__kernel_loff_t)); + +#ifdef __NR_sendfile64 + /* 32-bit applications use sendfile64 so offset is treated as a __kernel_loff_t */ + return __nolibc_syscall4(__NR_sendfile64, out_fd, in_fd, offset, count); +#else + __nolibc_static_assert(sizeof(*offset) == sizeof(__kernel_off_t)); + + return __nolibc_syscall4(__NR_sendfile, out_fd, in_fd, offset, count); +#endif +} + +static __attribute__((unused)) +ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count) +{ + return __sysret(_sys_sendfile(out_fd, in_fd, offset, count)); +} + +#endif /* _NOLIBC_SYS_SENDFILE_H */ diff --git a/tools/include/nolibc/types.h b/tools/include/nolibc/types.h index 8f3cb18df7f1c8..4373bd8a5b5f89 100644 --- a/tools/include/nolibc/types.h +++ b/tools/include/nolibc/types.h @@ -119,7 +119,7 @@ struct timeval { #define WEXITSTATUS(status) (((status) & 0xff00) >> 8) #define WIFEXITED(status) (((status) & 0x7f) == 0) #define WTERMSIG(status) ((status) & 0x7f) -#define WIFSIGNALED(status) ((status) - 1 < 0xff) +#define WIFSIGNALED(status) ((status) - 1U < 0xff) /* standard exit() codes */ #define EXIT_SUCCESS 0 diff --git a/tools/include/nolibc/unistd.h b/tools/include/nolibc/unistd.h index a264a20da13dd1..939d4e05a55d35 100644 --- a/tools/include/nolibc/unistd.h +++ b/tools/include/nolibc/unistd.h @@ -52,7 +52,11 @@ int access(const char *path, int amode) static __attribute__((unused)) int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1) { +#if defined(__NOLIBC_PAD_64BIT_SYSCALL_ARGUMENT_PAIR) + return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1); +#else return __nolibc_syscall3(__NR_ftruncate64, fd, length0, length1); +#endif } #define _sys_ftruncate64 _sys_ftruncate64 #endif diff --git a/tools/include/uapi/linux/coredump.h b/tools/include/uapi/linux/coredump.h index dc3789b78af021..6d0c53b534ea8d 100644 --- a/tools/include/uapi/linux/coredump.h +++ b/tools/include/uapi/linux/coredump.h @@ -11,12 +11,53 @@ * @COREDUMP_USERSPACE: userspace writes coredump * @COREDUMP_REJECT: don't generate coredump * @COREDUMP_WAIT: wait for coredump server + * @COREDUMP_RECORDS: send the coredump as a sequence of records instead of + * as a plain byte stream, see struct coredump_record_header; + * requires COREDUMP_KERNEL + * @COREDUMP_SPARSE: describe the holes in the coredump as zero records + * instead of transferring them; requires COREDUMP_RECORDS + * @COREDUMP_MEMORY_TYPES: dump the memory types in + * coredump_ack->memory_types instead of the ones + * the task selected; requires COREDUMP_KERNEL */ enum { COREDUMP_KERNEL = (1ULL << 0), COREDUMP_USERSPACE = (1ULL << 1), COREDUMP_REJECT = (1ULL << 2), COREDUMP_WAIT = (1ULL << 3), + COREDUMP_RECORDS = (1ULL << 4), + COREDUMP_SPARSE = (1ULL << 5), + COREDUMP_MEMORY_TYPES = (1ULL << 6), +}; + +/** + * coredump memory types + * @COREDUMP_MEMORY_ANON_PRIVATE: anonymous private memory + * @COREDUMP_MEMORY_ANON_SHARED: anonymous shared memory + * @COREDUMP_MEMORY_FILE_PRIVATE: file-backed private memory + * @COREDUMP_MEMORY_FILE_SHARED: file-backed shared memory + * @COREDUMP_MEMORY_ELF_HEADERS: the first page of a file-backed private + * mapping that starts an ELF file + * @COREDUMP_MEMORY_HUGETLB_PRIVATE: hugetlb private memory + * @COREDUMP_MEMORY_HUGETLB_SHARED: hugetlb shared memory + * @COREDUMP_MEMORY_DAX_PRIVATE: DAX private memory + * @COREDUMP_MEMORY_DAX_SHARED: DAX shared memory + * + * A bitmask of memory types a coredump may request to be included. New + * memory type bits must ensure that they do not steal memory from an + * existing one so a coredump server will continue to get the same + * coredumps even if a new bit is introduced. + */ +enum { + COREDUMP_MEMORY_ANON_PRIVATE = (1ULL << 0), + COREDUMP_MEMORY_ANON_SHARED = (1ULL << 1), + COREDUMP_MEMORY_FILE_PRIVATE = (1ULL << 2), + COREDUMP_MEMORY_FILE_SHARED = (1ULL << 3), + COREDUMP_MEMORY_ELF_HEADERS = (1ULL << 4), + COREDUMP_MEMORY_HUGETLB_PRIVATE = (1ULL << 5), + COREDUMP_MEMORY_HUGETLB_SHARED = (1ULL << 6), + COREDUMP_MEMORY_DAX_PRIVATE = (1ULL << 7), + COREDUMP_MEMORY_DAX_SHARED = (1ULL << 8), }; /** @@ -24,17 +65,19 @@ enum { * @size: size of struct coredump_req * @size_ack: known size of struct coredump_ack on this kernel * @mask: supported features + * @memory_types: the memory types the task selected + * @memory_types_mask: the memory types this kernel knows * * When a coredump happens the kernel will connect to the coredump * socket and send a coredump request to the coredump server. The @size * member is set to the size of struct coredump_req and provides a hint * to userspace how much data can be read. Userspace may use MSG_PEEK to * peek the size of struct coredump_req and then choose to consume it in - * one go. Userspace may also simply read a COREDUMP_ACK_SIZE_VER0 + * one go. Userspace may also simply read a COREDUMP_REQ_SIZE_VER0 * request. If the size the kernel sends is larger userspace simply * discards any remaining data. * - * The coredump_req->mask member is set to the currently know features. + * The coredump_req->mask member is set to the currently known features. * Userspace may only set coredump_ack->mask to the bits raised by the * kernel in coredump_req->mask. * @@ -42,15 +85,27 @@ enum { * struct coredump_ack the kernel knows. Userspace may only send up to * coredump_req->size_ack bytes to the kernel and must set * coredump_ack->size accordingly. + * + * @memory_types is set to the default memory types that are included in + * the coredump. This can be overridden by raising bits in + * coredump_ack->memory_types. + * + * @memory_types_mask contains a bitmask of all memory types the kernel + * knows about. A coredump server may only raise bits in + * coredump_ack->memory_types that are raised in + * coredump_req->memory_types_mask. */ struct coredump_req { __u32 size; __u32 size_ack; __u64 mask; + __u64 memory_types; + __u64 memory_types_mask; }; enum { COREDUMP_REQ_SIZE_VER0 = 16U, /* size of first published struct */ + COREDUMP_REQ_SIZE_VER1 = 32U, /* memory_types and memory_types_mask added */ }; /** @@ -58,6 +113,8 @@ enum { * @size: size of the struct * @spare: unused * @mask: features kernel is supposed to use + * @memory_types: memory types to dump, only with COREDUMP_MEMORY_TYPES + * in @mask * * The @size member must be set to the size of struct coredump_ack. It * may never exceed what the kernel returned in coredump_req->size_ack @@ -67,15 +124,30 @@ enum { * The @mask member must be set to the features the coredump server * wants the kernel to use. Only bits the kernel returned in * coredump_req->mask may be set. + * + * If COREDUMP_MEMORY_TYPES is raised in @mask the kernel dumps the + * memory types set in the @memory_types mask. Zero is valid and dumps + * no memory apart from the mappings that are always dumped. + * + * Note that memory a task excluded via MADV_DONTDUMP is always left + * out. A coredump server wanting to add or drop memory types instead of + * outright replacing it should simply copy coredump_req->memory_types + * and then mask off or raise types as needed. + * + * Note that @memory_types must be zero if COREDUMP_MEMORY_TYPES isn't + * raised. COREDUMP_MEMORY_TYPES requires COREDUMP_KERNEL and an ack of + * at least COREDUMP_ACK_SIZE_VER1 bytes. */ struct coredump_ack { __u32 size; __u32 spare; __u64 mask; + __u64 memory_types; }; enum { COREDUMP_ACK_SIZE_VER0 = 16U, /* size of first published struct */ + COREDUMP_ACK_SIZE_VER1 = 24U, /* memory_types added */ }; /** @@ -83,11 +155,12 @@ enum { * * The kernel will place a single byte on the coredump socket. The * markers notify userspace whether the coredump ack succeeded or - * failed. + * failed. After any marker other than COREDUMP_MARK_REQACK the kernel + * closes the connection and no coredump is generated. * * @COREDUMP_MARK_MINSIZE: the provided coredump_ack size was too small * @COREDUMP_MARK_MAXSIZE: the provided coredump_ack size was too big - * @COREDUMP_MARK_UNSUPPORTED: the provided coredump_ack mask was invalid + * @COREDUMP_MARK_UNSUPPORTED: the provided coredump_ack mask or memory types were invalid * @COREDUMP_MARK_CONFLICTING: the provided coredump_ack mask has conflicting options * @COREDUMP_MARK_REQACK: the coredump request and ack was successful * @__COREDUMP_MARK_MAX: the maximum coredump mark value @@ -101,4 +174,72 @@ enum coredump_mark { __COREDUMP_MARK_MAX = (1U << 31), }; +/** + * enum coredump_record_type - Type of a coredump record + * + * @COREDUMP_RECORD_DATA: the header is followed by ->len bytes of data + * @COREDUMP_RECORD_END: the coredump ends here, the header is not followed + * by any data and no further record is sent + * @COREDUMP_RECORD_ZERO: the header stands for ->len zero bytes and is not + * followed by any data + * @__COREDUMP_RECORD_TYPE_MAX: the maximum coredump record type value + */ +enum coredump_record_type { + COREDUMP_RECORD_DATA = 0U, + COREDUMP_RECORD_END = 1U, + COREDUMP_RECORD_ZERO = 2U, + __COREDUMP_RECORD_TYPE_MAX = (1U << 31), +}; + +/** + * struct coredump_record_header - header of a coredump record + * @size: size of struct coredump_record_header + * @type: one of enum coredump_record_type + * @flags: modifiers for this record + * @offset: offset in the coredump this record starts at + * @len: number of coredump bytes this record accounts for + * + * If the coredump server raises COREDUMP_RECORDS in coredump_ack->mask + * the kernel doesn't send the coredump as a plain byte stream. It sends + * a sequence of records instead. A COREDUMP_RECORD_DATA record is + * followed by @len bytes of actual coredump data. A + * COREDUMP_RECORD_ZERO record is followed by nothing and stands for + * @len zero bytes. A server that didn't raise COREDUMP_SPARSE never + * sees a zero record. Records arrive in order and leave no gaps. So + * @offset is the sum of the @len of all records before it. + * + * The last record is a COREDUMP_RECORD_END record. It is followed by + * nothing. Its @len is zero. Its @offset is the size of the coredump. + * The kernel only sends it once it has written the whole coredump. A + * server that hits end-of-file without having seen an end record must + * treat the coredump as incomplete. + * + * The @size member is set to the size of struct coredump_record_header + * the kernel knows and lets the header grow later. It comes first so it + * can be peeked. Userspace must consume @size bytes and discard + * anything beyond what it knows. It must refuse a @size smaller than + * COREDUMP_RECORD_HEADER_SIZE_VER0. @size covers the header alone. + * @offset and @len count coredump bytes. + * + * The @flags member carries modifiers that change how the record is to + * be interpreted. No flag is defined yet. Userspace must refuse a + * record carrying a flag or a type it doesn't know. Every new record + * type is raised in coredump_req->mask as a feature of its own. A + * server only ever sees the types it asked for. + * + * COREDUMP_RECORDS must be combined with COREDUMP_KERNEL, and + * COREDUMP_SPARSE with COREDUMP_RECORDS. + */ +struct coredump_record_header { + __u32 size; + __u32 type; + __u64 flags; + __u64 offset; + __u64 len; +}; + +enum { + COREDUMP_RECORD_HEADER_SIZE_VER0 = 32U, /* size of first published struct */ +}; + #endif /* _UAPI_LINUX_COREDUMP_H */ diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py index 0b1ffdf9f7a35e..7e94635bf7d94a 100755 --- a/tools/lib/python/jobserver.py +++ b/tools/lib/python/jobserver.py @@ -29,6 +29,10 @@ The end goal here is to keep the total number of build tasks under the limit established by the initial ``make -j$n_proc`` call. +A command that takes its thread count on the command line rather than from +the environment can use the ``%PARALLELISM%`` token in its arguments, which +is replaced by the count, or by 1 when there is no jobserver. + See: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver """ @@ -38,6 +42,8 @@ import subprocess import sys +PARALLELISM_TOKEN = "%PARALLELISM%" + def warn(text, *args): print(f'WARNING: {text}', *args, file = sys.stderr) @@ -182,6 +188,9 @@ def run(self, cmd, *args, **pwargs): Run a command setting PARALLELISM env variable to the number of available job slots (claim) + 1, e.g. it will reserve claim slots to do the actual build work, plus one to monitor its children. + + Any %PARALLELISM% in the command's arguments is replaced by the same + number, or by 1 when there is no jobserver. """ self.open() # Ensure that self.claim is set @@ -192,4 +201,8 @@ def run(self, cmd, *args, **pwargs): if self.claim: os.environ["PARALLELISM"] = str(self.claim) + if not isinstance(cmd, str): + parallelism = str(self.claim or 1) + cmd = [arg.replace(PARALLELISM_TOKEN, parallelism) for arg in cmd] + return subprocess.call(cmd, *args, **pwargs) diff --git a/tools/lib/python/kdoc/c_lex.py b/tools/lib/python/kdoc/c_lex.py index cb95f51724483d..efbc23c01abdfe 100644 --- a/tools/lib/python/kdoc/c_lex.py +++ b/tools/lib/python/kdoc/c_lex.py @@ -68,7 +68,7 @@ class CToken(): MISMATCH = 255 #: an error indicator: should never happen in practice. - # Dict to convert from an enum interger into a string. + # Dict to convert from an enum integer into a string. _name_by_val = {v: k for k, v in dict(vars()).items() if isinstance(v, int)} # Dict to convert from string to an enum-like integer value. diff --git a/tools/lib/python/kdoc/kdoc_parser.py b/tools/lib/python/kdoc/kdoc_parser.py index d9ad1ddc87dd13..80d4245acd1725 100644 --- a/tools/lib/python/kdoc/kdoc_parser.py +++ b/tools/lib/python/kdoc/kdoc_parser.py @@ -831,6 +831,13 @@ def dump_struct(self, ln, proto, source): # source = source proto = trim_private_members(proto) + + # + # Expand macro-based declarations, like context_lock_struct(), + # into a plain struct/union declaration before splitting it, + # as dump_var() and dump_function() already do. + # + proto = self.xforms.apply("struct", proto) struct_parts = self.split_struct_proto(proto) if not struct_parts: self.emit_msg(ln, f"{proto} error: Cannot parse struct or union!") @@ -841,10 +848,6 @@ def dump_struct(self, ln, proto, source): self.emit_msg(ln, f"expecting prototype for {decl_type} {self.entry.identifier}. " f"Prototype was for {decl_type} {declaration_name} instead") return - # - # Go through the list of members applying all of our transformations. - # - members = self.xforms.apply("struct", members) # # Deal with embedded struct and union members, and drop enums entirely. diff --git a/tools/lib/python/kdoc/xforms_lists.py b/tools/lib/python/kdoc/xforms_lists.py index e3dda2fe8a5371..320f977c5bac2c 100644 --- a/tools/lib/python/kdoc/xforms_lists.py +++ b/tools/lib/python/kdoc/xforms_lists.py @@ -45,6 +45,7 @@ class CTransforms: (CMatch("DECLARE_HASHTABLE"), r"unsigned long \1[1 << ((\2) - 1)]"), (CMatch("DECLARE_KFIFO"), r"\2 *\1"), (CMatch("DECLARE_KFIFO_PTR"), r"\2 *\1"), + (CMatch("DECLARE_PIDS"), r"struct pid *\1[(\2) + 1]"), (CMatch("(?:__)?DECLARE_FLEX_ARRAY"), r"\1 \2[]"), (CMatch("DEFINE_DMA_UNMAP_ADDR"), r"dma_addr_t \1"), (CMatch("DEFINE_DMA_UNMAP_LEN"), r"__u32 \1"), @@ -53,7 +54,7 @@ class CTransforms: (CMatch("__attribute__"), ""), # - # Macro __struct_group() creates an union with an anonymous + # Macro __struct_group() creates a union with an anonymous # and a non-anonymous struct, depending on the parameters. We only # need one of those at kernel-doc, as we won't be documenting the same # members twice. @@ -62,6 +63,7 @@ class CTransforms: (CMatch("struct_group_attr"), r"struct { \3+ };"), (CMatch("struct_group_tagged"), r"struct { \3+ };"), (CMatch("__struct_group"), r"struct { \4+ };"), + (CMatch("context_lock_struct"), r"struct \1"), ] #: Transforms for function prototypes. diff --git a/tools/objtool/Documentation/klp-test-design.txt b/tools/objtool/Documentation/klp-test-design.txt new file mode 100644 index 00000000000000..2082c277197f4e --- /dev/null +++ b/tools/objtool/Documentation/klp-test-design.txt @@ -0,0 +1,286 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================================== +Design of the objtool klp test harness +====================================== + +tools/objtool/tests/ holds unit tests for the klp subcommands of +objtool -- ``klp checksum``, ``klp diff``, ``klp post-link`` and +``--klp-symids`` -- which together turn two builds of the kernel into a +livepatch module. + +This document explains how the harness is built and why. For the rules to +follow when adding a test, see klp-write-tests.txt. + + +TL;DR +===== + +One run covers one compiler and one architecture; CI runs the combinations. +Build objtool first -- it needs libelf and libxxhash -- and the same ARCH is +used for both steps. + +Natively, with gcc:: + + make -C tools/objtool + make -C tools/objtool tests + +Natively, with clang -- LLVM=1 additionally selects the LLVM binutils:: + + CC=clang make -C tools/objtool tests + LLVM=1 make -C tools/objtool tests + +Cross, with gcc -- an arm64 host running the x86 tests:: + + ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool + ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests + +Cross, with clang. It defaults to the host triple however it is invoked, so +--target= is what makes it emit x86; OBJCOPY is needed because BFD's is +usually built for the host's target alone:: + + ARCH=x86_64 make -C tools/objtool + ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" OBJCOPY=llvm-objcopy \ + make -C tools/objtool tests + +A run ends with a totals line; anything other than fail:0 is a real result:: + + # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 + +Useful extras:: + + tools/objtool/tests/run-tests.sh basic # one test, by name + tools/objtool/tests/run-tests.sh --keep basic # and keep what it built + + +Why unit tests are possible at all +================================== + +klp-build is a pipeline: build the kernel twice, checksum both, diff them, +link the result. Testing that end to end means two kernel builds per case, +which is too slow to run often and too heavy to keep in the tree. + +Three properties make a much cheaper test possible. + +**objtool has no configuration-dependent logic.** It never reads ``.config``. +Every ``CONFIG_`` string in its source is a comment or one error message, and +its only build-time conditionals are driven by host libraries and the target +architecture. Configuration reaches objtool through exactly two channels: the +``objtool-args-$(CONFIG_*)`` lines in scripts/Makefile.lib, and the +contents of the object handed to it. + +**The klp subcommands use none of the first channel.** Of objtool's options +they consult three -- ``checksum``, ``debug_checksum``, ``dryrun`` -- all from +their own command line. So klp behaviour varies with configuration *only* +through the input object. + +**Therefore a test can reproduce any configuration's behaviour by reproducing +its input.** Compile a small freestanding fixture with the flags that +configuration would have used, and objtool cannot tell the difference. No +kernel, no ``.config``, no object cache. + +The whole suite runs in a few seconds. + + +Shape of a test +=============== + +Each test compiles one fixture twice -- once plain, once with ``-DPATCHED`` -- +runs ``klp checksum`` over both, diffs them, and asserts on properties of the +output object:: + + . "$(dirname "$0")/../lib.sh" + + setup + build_pair basic.c + + assert_input_symbol changed + run_diff + + assert_patched changed + assert_not_patched untouched + + pass "changed function cloned, unchanged function left alone" + +Assertions check properties, never recorded output. Codegen varies between +compilers and versions, so a golden file would report churn rather than +regressions. + + +Layout +====== + +:: + + tools/objtool/tests/ + lib.sh the harness: everything a test may call + run-tests.sh selects, runs and classifies + generic/ + test-*.sh + fixtures/*.c + x86/ + test-*.sh + fixtures/*.c + +Which architecture a test is for is expressed by where it lives. The runner +executes ``generic/`` plus the directory matching this architecture, so a test +which cannot apply is not run rather than running in order to report that it +did not. There is no ``x86_only`` helper, and no lookup letting an +architecture fixture shadow a generic one: an architecture-specific test +carries its own fixtures. + +Compilers cannot be expressed the same way, because CI varies ``CC`` over the +same tree. A compiler requirement stays a declaration inside the test +(``gcc_only``, ``clang_only``). + + +The environment is established once +=================================== + +Sourcing lib.sh runs ``klp_preflight``, which checks that objtool +exists and has klp support, that ``$CC`` works, that the binutils are present, +and which architecture this is. The answers are exported, so: + +* ``run-tests.sh`` sources lib.sh too, and therefore knows the + architecture before it chooses which tests to run; +* each test inherits the answers rather than repeating the work; +* a test run on its own establishes them for itself. + +Preflight answers only whether the suite can run at all. A suite which cannot +run must not exit 0 looking like one which passed, so a missing objtool fails +the whole run with a TAP ``Bail out!`` rather than skipping each test in turn. +What a *particular* compiler can do is a different question, left to the test +which cares. + + +Outcomes +======== + +Output is TAP. The distinction the harness cares most about is between kinds +of skip, because a skip is how a suite quietly stops testing anything: + +``declared`` + The test said in advance it does not apply -- ``gcc_only`` on a clang run. + Expected indefinitely. + +``probe`` + The construct did not turn up in the built object this time. Weaker: one + which becomes permanent is a fixture that has stopped testing anything. + +``undeclared`` + Counted as a **failure**. A test which gives up for a reason it never + declared is a hole, not an outcome. + +``xfail``/``xpass`` come with them, so a known failure is reported rather than +commented out, and one which starts passing says so instead of going quietly +green. + +The runner classifies the TAP result line, not everything a test printed: +objtool warns on stderr and that output is captured, so a stray line ahead of +the result would otherwise leave the exit status to decide -- and an expected +failure exits 0. + +A run ends with a totals line:: + + # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 + +and reports what it left out:: + + # not run: 5 tests in x86/ (this run is arm64) + + +Working directories +=================== + +A run gets one directory; each test gets a subdirectory of it, mirroring the +source layout:: + + /tmp/klp-tests.XXXXXXXX/ + generic/test-basic/{orig.o,patched.o,out.o,Module.symvers,...} + x86/test-kcfi/... + +By default (``KEEP=failed``) only failing tests keep their directories; the +runner reports where they are. ``KEEP=all`` keeps every test's directory; +``KEEP=none`` removes them all. The runner ``rmdir``s the run directory when +it is empty -- which fails if anything was left behind unexpectedly, so a test +which dies without cleaning up is reported rather than silently leaking. + + +Running +======= + +:: + + make -C tools/objtool # needs libelf and libxxhash + make -C tools/objtool tests + + CC=clang make -C tools/objtool tests # the other toolchain + LLVM=1 make -C tools/objtool tests # and its binutils too + + make -C tools/objtool tests KEEP=all # keep every test's workdir + make -C tools/objtool tests KEEP=none # remove all workdirs + + tools/objtool/tests/run-tests.sh basic # one test; failures kept by default + +A run covers one compiler and one architecture; CI runs the combinations. + +Cross-compiled runs +------------------- + +objtool klp is built only where ARCH_HAS_KLP is set, which today means x86 -- +so an arm64 machine cannot run any of this natively. It can run all of it +cross, because objtool is a host tool that only reads and rewrites ELF, and +the tests only compile fixtures and inspect the objects. Nothing has to +execute target code. + +:: + + ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool + ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests + +objtool itself stays a native binary: it is built with HOSTCC, not CC, so +setting a cross compiler cannot produce one the host is unable to run. ARCH +selects both the objtool target and the directory of tests to run. + +clang needs telling, since it defaults to the host triple however it is +invoked. LLVM=1 with CROSS_COMPILE does that for you -- the --target= it +derives is forwarded to the tests -- and the fixtures include no kernel +headers, so no sysroot is needed: + +:: + + ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- LLVM=1 \ + make -C tools/objtool tests + +Naming the compiler by hand works too, and is what to reach for when the +target triple is not the one CROSS_COMPILE implies: + +:: + + ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" \ + OBJCOPY=llvm-objcopy make -C tools/objtool tests + +CROSS_COMPILE picks the binutils, and each can be overridden on its own. +readelf reads any target and rarely needs overriding; BFD's objcopy is usually +built for the host's alone, hence OBJCOPY=llvm-objcopy above, or install +binutils-multiarch. + +Either readelf will do. The assertions read readelf's output, and the two +spell some of it differently -- GNU prints "OS [0xff20]" for SHN_LIVEPATCH +where llvm-readelf prints "OS[0xff20]" -- so they accept both. + +Getting this wrong is easy and the harness refuses rather than producing a +misleading result. "CC=clang ARCH=x86_64" alone selects the x86 tests and +then builds arm64 objects; preflight compiles a probe object, hands it to +objtool, and stops the run if they disagree about the architecture, or if +ARCH does not match what the compiler emits. + + +What this does not cover +======================== + +These are unit tests for objtool's klp subcommands. They do not build a +kernel, do not run scripts/livepatch/klp-build, and do not load a +livepatch. Behaviour which only appears when the kernel applies a patch -- +the module loader refusing a relocation, late module patching ordering -- has +to be tested by booting, and is out of scope here. diff --git a/tools/objtool/Documentation/klp-write-tests.txt b/tools/objtool/Documentation/klp-write-tests.txt new file mode 100644 index 00000000000000..eb2cacaf572079 --- /dev/null +++ b/tools/objtool/Documentation/klp-write-tests.txt @@ -0,0 +1,266 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +Writing a test for objtool's klp code +===================================== + +Instructions for adding a test to tools/objtool/tests/. Read +klp-test-design.txt first if you need to know how the harness works; this +document is the procedure and the rules. + +Two kinds of request bring you here: + +* *"write a test for commit "* -- a fix went in without one. +* *"port the test at , written against another harness"* -- a case + exists elsewhere and should live in tree. + +Both follow the same procedure. + + +The one rule that matters +========================= + +**A test is not finished until you have watched it fail.** + +Break the thing it guards -- revert the fix, or sabotage the exact line -- and +confirm the test fails. Then restore and confirm it passes. A test that has +never failed is not known to test anything, and this suite has produced +several that passed against deliberately broken code: + +* an alternatives fixture whose empty entry pointed at its own end label rather + than the neighbour's replacement, so the bug it guarded made no difference; +* a sympos fixture where symbol-table order and address order agreed, so + counting and reading the linked image gave the same answer; +* a string fixture using a named ``char[]``, which never reached the + contents-hashing path because that keys on ``SHF_STRINGS``; +* a static array the compiler proved constant, folded to zero, and emitted no + relocation for -- so the two builds were byte-identical. + +Every one looked correct. Say in the commit message how you verified, and if +you could not isolate the behaviour to a single line, **say that too** rather +than implying otherwise. + + +Procedure +========= + +1. **Read the fix.** What input reaches the broken line? What is observable + in the output object when it misbehaves -- a missing section, a relocation + naming the wrong symbol, an unchanged checksum, a rejected build? If + nothing is observable, stop and say so; see `When to give up`_. + +2. **Decide where it lives.** ``generic/`` unless the fixture needs + architecture-specific assembly or the behaviour is architecture-specific, + in which case ``x86/`` (or a new directory named for the architecture). + +3. **Write the fixture** in the same directory's ``fixtures/``. Reuse an + existing one if it already produces the shape; add a ``-D`` knob rather + than copying a fixture to change one line. + +4. **Write the test.** Assert the *premise* before the result -- see + `State the premise`_. + +5. **Verify by breaking the code.** Then restore. + +6. **Run the whole suite under both compilers**:: + + make -C tools/objtool tests + CC=clang make -C tools/objtool tests + +7. **Commit** the test and its fixture together, alone. One test per commit. + + +Writing the fixture +=================== + +Fixtures are freestanding C. No kernel headers -- write out the kernel +structure by hand if you need one, as the existing special-section fixtures do. + +Every fixture needs a ``.modinfo`` name, because klp diff reads the object's +module name from it:: + + static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +Use ``MODNAME`` if the test needs to vary it. Note that most fixtures hardcode +``vmlinux``: passing ``-DMODNAME`` to one that does silently does nothing and +the test quietly becomes a vmlinux test. + +The patched build is selected with ``-DPATCHED``. For a fixture with several +variants, gate each on both, so the original is always the baseline:: + + #if defined(PATCHED) && defined(WHICH_CALL) + r = callee_b(x); + #else + r = callee_a(x); + #endif + +and select one per build: ``build_pair foo.c -DWHICH_CALL``. Without the +``defined(PATCHED)`` the flag applies to *both* builds and nothing differs. + +Traps that have bitten before +----------------------------- + +* **The compiler optimises your fixture away.** A static never written is + proved constant, its reads folded, and no relocation emitted. Add a writer + the compiler cannot see through. +* **String literals versus named arrays.** The contents-hashing path keys on + ``SHF_STRINGS``, which the compiler sets on the mergeable section a *literal* + lands in, not on a ``char[]`` given a section of its own. +* **Per-function sections hide movement.** With the default + ``-ffunction-sections`` every function sits at offset 0 of its own section, + so nothing ever moves. A test about position needs + ``build_pair foo.c -fno-function-sections``. +* **Special sections need boundaries.** Either an entsize on the section or an + ``ANNOTATE_DATA_SPECIAL`` annotation, or klp diff reports "missing special + section entsize or annotations". Their targets need real (global) symbols, + or it reports "failed to convert reloc sym". +* **Prefer letting objtool generate what objtool generates.** + ``.static_call_sites``, ``.mcount_loc``, ``.ibt_endbr_seal`` and ORC come + from its check pass. Call ``run_objtool_check --mcount`` and let it build + them; a hand-written copy tests your reading of the format, not the format. + + Write one by hand only when the test needs a shape objtool will not produce, + and say so in the fixture. generic/fixtures/static_call.c does: objtool + emits the site but not the ``ANNOTATE_DATA_SPECIAL`` that describes its + boundaries -- those come from the kernel's macros -- so a fixture which has + to vary whether the annotation is there writes both itself. + + +Writing the test +================ + +Start from the shortest existing test, generic/test-basic.sh. + +State the premise +----------------- + +A test which asserts only on the output passes when the compiler never emitted +the construct in the first place, and reads as coverage it does not have. Say +what the input must contain:: + + assert_input_section __jump_table # the fixture must produce it -> fail + require_input_section .kcfi_traps # this compiler may not -> skip + +Prefer ``assert_*``. Reach for ``require_*`` only where absence genuinely +depends on compiler version or flags, and follow it with something +unconditional so the test can never be entirely vacuous. + +Where a test would otherwise duplicate a sibling, assert what makes it +different. ``test-jump-label-module-static-key`` checks that the key really is +reached through its section symbol -- without that it is a second copy of +``test-jump-label-module-key``. + +Assert both directions +---------------------- + +Check that the right thing happened *and* that the wrong thing did not. A klp +diff which clones everything is as wrong as one which clones nothing:: + + assert_patched changed + assert_not_patched untouched + +Skips +----- + +* ``gcc_only``/``clang_only`` -- a settled fact about the compiler. Declared, + so it reads as expected forever. +* ``probe_skip`` -- this toolchain did not produce the construct. Include what + to do about it if there is anything:: + + probe_skip "no matching clang/lld pair for a ThinLTO link; set THIN_CC and THIN_LD to one" + +* A bare ``skip`` is **counted as a failure**. Never use it. + +Standing in for a kernel configuration +-------------------------------------- + +``FIXTURE_CFLAGS`` is what a fixture is built with. Since objtool reads no +``.config``, changing these flags is how a test covers a configuration without +building a kernel. Two ways: + +* trailing arguments to ``build_pair``/``build_one``, which win, and cover + anything expressible as a negation:: + + build_pair foo.c -fno-function-sections + +* otherwise assign ``FIXTURE_CFLAGS`` before building. + +Either way **say in a comment which kernel configuration the change stands in +for**. A flag with no stated motive is indistinguishable from a mistake. + + +What the test's comment must say +================================ + +The comment at the top is the test's justification. It should let a reader +decide, without archaeology, whether a skip or a failure matters. Include: + +* **what breaks** in the running kernel if the behaviour regresses -- not the + mechanism, the consequence; +* **why it is not caught otherwise**, which is usually "nothing fails at build + time"; +* **the fix commit** it guards, if there is one; +* **anything load-bearing about the fixture** that is not obvious, especially + anything you got wrong first. + +That last point is the one people skip. If the fixture has to be built without +per-function sections, or the static must not be named ``__warned``, or the key +must be file-local -- write it down, or the next person will simplify it away. + + +Porting a test from another harness +=================================== + +Read the original's *case*, not its code. The other harness probably builds a +real kernel module; here you write freestanding C. A transliteration will +usually test something else. + +* Work out which objtool behaviour the case exercises, then produce that shape + the cheapest way here. +* Verify by breaking the code, exactly as for a new test -- a port is not + correct because the original was. +* If the original names a fix commit, cite it. +* Credit the source in the commit message with the trailers the original + carried, followed by your own. + +Sometimes the port shows the case is already covered, and sometimes it shows +the case cannot be reproduced here. Both are results; report them rather than +committing something that passes vacuously. + + +When to give up +=============== + +Some behaviour cannot be reached from a compiled fixture. Say so, with what +you tried, instead of committing a test that passes either way. Examples that +were genuinely abandoned: + +* a memory leak -- needs valgrind, not an assertion on ELF; +* ``mkstemp`` with long paths, and other I/O edge cases; +* a NULL dereference reachable only through a debug path; +* changes made redundant by a fallback: removing the code changes no output + because something else already handles the case; +* a fix whose code has since been rewritten, so there is nothing left to + revert. + +Also stop when the behaviour depends on something outside the fixture's +control -- an ELF library's handling of empty sections, or a compiler version's +naming of anonymous data. A test which passes for you and skips for everyone +else is worse than none. + + +Checklist +========= + +Before committing: + +* the test fails with the code broken, and passes with it fixed +* the whole suite passes under **both** gcc and clang +* the premise is asserted, not assumed +* both directions are asserted where that applies +* no bare ``skip`` +* the fixture is in the same directory as the test +* the comment names the consequence, the fix commit, and anything load-bearing +* the commit contains one test and its fixtures, and nothing else +* the commit message says how you verified it diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile index 4cc2e756af840b..db99c4759b2c3b 100644 --- a/tools/objtool/Makefile +++ b/tools/objtool/Makefile @@ -153,6 +153,19 @@ clean: $(LIBSUBCMD)-clean mrproper: clean $(call QUIET_CLEAN, objtool) $(RM) $(OBJTOOL) +# The kernel's own Makefile sets OBJCOPY, to llvm-objcopy under LLVM and to +# $(CROSS_COMPILE)objcopy otherwise; tools/scripts/Makefile.include names only +# LLVM_OBJCOPY and leaves OBJCOPY unset. Forwarding it unset would hand the +# tests an empty string, and they would fall back to GNU objcopy even when +# asked for an LLVM toolchain. +OBJCOPY ?= $(if $(LLVM),$(LLVM_OBJCOPY),$(CROSS_COMPILE)objcopy) + +tests: $(OBJTOOL) + $(Q)OBJTOOL=$(abspath $(OBJTOOL)) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \ + CC='$(CC) $(CLANG_CROSS_FLAGS)' LD='$(LD)' \ + READELF='$(READELF)' OBJCOPY='$(OBJCOPY)' \ + KEEP=$(KEEP) $(srctree)/tools/objtool/tests/run-tests.sh + FORCE: -.PHONY: clean mrproper FORCE +.PHONY: clean mrproper tests FORCE diff --git a/tools/objtool/tests/generic/fixtures/abs_and_addressable.c b/tools/objtool/tests/generic/fixtures/abs_and_addressable.c new file mode 100644 index 00000000000000..6392ff99af421d --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/abs_and_addressable.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Two constructs which appear all over the kernel and must not upset klp + * checksum or klp diff. + * + * An absolute symbol (SHN_ABS) has no section, so anything walking sym->sec + * without checking dereferences NULL. The kernel makes them with linker + * scripts and with .set in asm; VDSO and the fixed-address per-cpu bases are + * the usual sources. + * + * __ADDRESSABLE() emits a pointer into .discard.addressable purely to keep a + * symbol referenced. It is discarded at link time and means nothing to a + * livepatch, but the pointer is a relocation like any other and has to survive + * being looked at. + * + * Neither is the subject of the patch; the point is that their presence does + * not disturb the function that is. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* SHN_ABS, referenced from code. */ +extern char abs_sym[]; +__asm__(".globl abs_sym\n" + ".set abs_sym, 0x1234\n"); + +int helper(int x); +int helper(int x) { return x + 1; } + +/* The shape of __ADDRESSABLE(helper). */ +__asm__(".pushsection .discard.addressable, \"aw\"\n" + ".balign 8\n" + ".quad helper\n" + ".popsection\n"); + +int target(int x) +{ +#ifdef PATCHED + return helper(x) + (int)(long)abs_sym + 1; +#else + return helper(x) + (int)(long)abs_sym; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/basic.c b/tools/objtool/tests/generic/fixtures/basic.c new file mode 100644 index 00000000000000..811529e7bfb1d9 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/basic.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* One changed function and one unchanged function. */ + +/* klp diff takes the object's module name from .modinfo */ +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int untouched(int x) +{ + return x * 3; +} + +int changed(int x) +{ +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/changed_data.c b/tools/objtool/tests/generic/fixtures/changed_data.c new file mode 100644 index 00000000000000..b52461835444ff --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/changed_data.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Data whose value differs between the two builds. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +#ifdef PATCHED +int klp_test_data = 2; +#else +int klp_test_data = 1; +#endif + +int target(int x) +{ + return x + klp_test_data; +} diff --git a/tools/objtool/tests/generic/fixtures/checksum_data.c b/tools/objtool/tests/generic/fixtures/checksum_data.c new file mode 100644 index 00000000000000..6310af5c02d348 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/checksum_data.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Data objects whose checksums must move for reasons the raw bytes do not + * show. + * + * checksum_update_object() hashes a data symbol's length and its bytes, and + * then walks its relocations: a reference into a string section contributes + * the string's *contents*, and any other reference contributes the target + * symbol's name and the adjusted addend. So three changes that leave the + * object's own bytes identical still have to change its checksum: + * + * Each variant is selected by a -D on the patched build only, so the original + * is always the baseline: + * + * WHICH_FUNC the function pointer points somewhere else + * WHICH_STR the string pointer points at a different literal + * STR_CONTENT the string it points at is edited in place + * WHICH_SLOT the same array, at a different index: addend only + * WHICH_PRIV likewise, but a static, reached through its section symbol + * + * The last is the interesting one. Nothing in the pointer changes -- same + * section, same offset -- so a checksum that hashed only the relocation and + * not what it referred to would call the object unchanged, and the patched + * kernel would keep the old string. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int callee_a(int x); +int callee_b(int x); +int callee_a(int x) { return x + 1; } +int callee_b(int x) { return x + 2; } + +/* + * String *literals*, not named arrays. The contents-hashing path keys on + * SHF_STRINGS, which the compiler sets on the mergeable .rodata.str1.1 a + * literal lands in and not on a named char[] given a section of its own. A + * fixture using the latter exercises the ordinary name-and-addend path and + * reports nothing when the text changes. + */ +#if defined(PATCHED) && defined(STR_CONTENT) +#define MESSAGE "edited" +#else +#define MESSAGE "original" +#endif + +/* A plain data object: only its own bytes decide the checksum. */ +#if defined(PATCHED) && defined(PLAIN_VALUE) +int plain = 43; +#else +int plain = 42; +#endif + +/* + * A .bss object, where length is the only thing there is to hash: the section + * has no data, so the bytes are skipped and only sym->len distinguishes this + * from an object of another size. An initialised array would not isolate it + * -- growing one changes the hashed bytes as well. + */ +#if defined(PATCHED) && defined(LONGER) +char sized[4]; +#else +char sized[2]; +#endif + +/* + * A reference into the middle of an array: same target symbol, different + * addend. Nothing else in the object changes, so this is the only way to see + * whether the addend is hashed at all. + */ +int slots[4]; + +/* + * A file-local array. A reference to a static lands on its section symbol + * plus an offset, so the hash has to resolve that back to the underlying + * object before it has a name to hash at all -- a different code path from the + * global above, and one that silently contributes nothing when it fails. + */ +static int priv_slots[4]; + +struct desc { + int (*fn)(int arg); + const char *str; + int *slot; + int *priv; +}; + +const struct desc descriptor = { +#if defined(PATCHED) && defined(WHICH_FUNC) + .fn = callee_b, +#else + .fn = callee_a, +#endif +#if defined(PATCHED) && defined(WHICH_STR) + .str = "a different literal", +#else + .str = MESSAGE, +#endif +#if defined(PATCHED) && defined(WHICH_SLOT) + .slot = &slots[2], +#else + .slot = &slots[1], +#endif +#if defined(PATCHED) && defined(WHICH_PRIV) + .priv = &priv_slots[3], +#else + .priv = &priv_slots[1], +#endif +}; + +int target(int x) +{ + return descriptor.fn(x) + plain + sized[0] + (int)descriptor.str[0] + + *descriptor.slot + *descriptor.priv; +} diff --git a/tools/objtool/tests/generic/fixtures/checksum_insn.c b/tools/objtool/tests/generic/fixtures/checksum_insn.c new file mode 100644 index 00000000000000..10f70a74a97639 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/checksum_insn.c @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Instruction operands whose change must move a function's checksum even + * though the instruction bytes themselves do not. + * + * checksum_update_insn() hashes the raw bytes and then, when the instruction + * carries a relocation, what that relocation refers to: a string section + * contributes the string's contents, anything else the target symbol's name + * and the adjusted addend. A reference to a static arrives as a section + * symbol and has to be resolved back to the object first. + * + * The bytes are identical in every case below -- a rel32 operand is zero in + * the object and supplied by the relocation -- so a checksum that stopped at + * the bytes would call all of these unchanged. + * + * Each variant applies to the patched build only: + * + * WHICH_CALL calls a different function + * STR_CONTENT passes a literal whose text was edited + * WHICH_SLOT reads a different index of a global array: addend only + * WHICH_PRIV the same, for a static, reached through its section symbol + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int callee_a(int x); +int callee_b(int x); +int sink(const char *s); + +int slots[4]; + +/* + * A file-local array, plus a writer the compiler cannot see through. Without + * one it can prove the array is never written, folds every read to zero, and + * emits no relocation at all -- so the reference this is here to exercise does + * not exist. + */ +static int priv_slots[4]; + +void set_priv(int i, int v); +void set_priv(int i, int v) +{ + priv_slots[i] = v; +} + +#if defined(PATCHED) && defined(STR_CONTENT) +#define MESSAGE "edited" +#else +#define MESSAGE "original" +#endif + +int target(int x) +{ + int r; + +#if defined(PATCHED) && defined(WHICH_CALL) + r = callee_b(x); +#else + r = callee_a(x); +#endif + + r += sink(MESSAGE); + +#if defined(PATCHED) && defined(WHICH_SLOT) + r += slots[2]; +#else + r += slots[1]; +#endif + +#if defined(PATCHED) && defined(WHICH_PRIV) + r += priv_slots[3]; +#else + r += priv_slots[1]; +#endif + + return r; +} diff --git a/tools/objtool/tests/generic/fixtures/checksum_position.c b/tools/objtool/tests/generic/fixtures/checksum_position.c new file mode 100644 index 00000000000000..4320bc22059226 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/checksum_position.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A function whose position in the section changes between the two builds, + * without the function itself changing. + * + * target() calls callee() twice, and a call within the same section needs no + * relocation: the displacement is in the instruction. It is that displacement + * which moves, and hashing those bytes makes the checksum move with it. Both + * must therefore share a section, which is why the test passes + * -fno-function-sections. + * + * What has to change is the distance between the two, and PATCHED changes it + * by aligning them rather than by inserting a function between them. Where a + * compiler puts an added function is its own business: gcc emits these in + * source order, so a function written between callee() and target() separates + * them, but clang emits target() immediately before callee() whatever the + * source says, and an added function lands ahead of both. That moves target() + * without moving it relative to callee(), the displacement comes out identical + * in both builds, and the test passes without having asked anything. + * + * Alignment moves the functions apart on both, and moves neither function's + * own instructions -- which is exactly the distinction under test. + */ + +#ifdef PATCHED +#define MOVED __attribute__((aligned(64))) +#else +#define MOVED +#endif + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +__attribute__((noinline)) MOVED static int callee(int x) +{ + return x * 5 + 1; +} + +__attribute__((noinline)) MOVED int target(int x) +{ + return callee(x) + callee(x + 1); +} diff --git a/tools/objtool/tests/generic/fixtures/checksum_skip.c b/tools/objtool/tests/generic/fixtures/checksum_skip.c new file mode 100644 index 00000000000000..973bdc10295df3 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/checksum_skip.c @@ -0,0 +1,47 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Symbols calculate_checksums() must not give an entry of their own. + * + * Three kinds are skipped, for two different reasons: + * + * zero-length there is nothing to hash, and an entry keyed on the + * symbol's address would collide with whatever really lives + * there. + * alias a second name for an address already checksummed. + * cold part hashed as part of its parent, which func_for_each_insn() + * walks into, so a separate entry would double-count it. + * + * An entry per address is the invariant: .discard.sym_checksum is looked up by + * the address a relocation points at, so two entries for one address make the + * lookup ambiguous and one of the two checksums unreachable. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* Zero-length: an object symbol of size 0, in a section of its own. */ +extern char empty_marker[]; +__asm__(".pushsection .data.empty_marker,\"aw\",@progbits\n" + ".globl empty_marker\n" + ".type empty_marker, @object\n" + "empty_marker:\n" + ".size empty_marker, 0\n" + ".popsection\n"); + +int real_function(int x); +int real_function(int x) +{ +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} + +/* Alias: a second name for real_function's address. */ +int alias_function(int x) __attribute__((alias("real_function"))); + +int target(int x) +{ + return real_function(x) + alias_function(x) + (int)(long)empty_marker; +} diff --git a/tools/objtool/tests/generic/fixtures/cold_function.c b/tools/objtool/tests/generic/fixtures/cold_function.c new file mode 100644 index 00000000000000..f6d410983ce238 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/cold_function.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Function the compiler may split into a hot part and a foo.cold part. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +static void __attribute__((cold, noinline)) slow_path(int x) +{ + __asm__ volatile("" :: "r"(x)); +} + +int target(int x) +{ + if (__builtin_expect(x < 0, 0)) + slow_path(x); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/cross_module.c b/tools/objtool/tests/generic/fixtures/cross_module.c new file mode 100644 index 00000000000000..c170bde0666f40 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/cross_module.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A function which calls out to another object. MODNAME selects which object + * this one is, so a test can make the caller a module and the callee's owner + * something else. + */ + +#ifndef MODNAME +#define MODNAME "vmlinux" +#endif + +/* klp diff takes the object's module name from .modinfo */ +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=" MODNAME; + +extern int other_mod_func(int x); + +int target(int x) +{ +#ifdef PATCHED + return other_mod_func(x) + 2; +#else + return other_mod_func(x) + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/data_alignment.c b/tools/objtool/tests/generic/fixtures/data_alignment.c new file mode 100644 index 00000000000000..900253dfb2cbb8 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/data_alignment.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Data with an alignment stricter than its size. + * + * A cloned data section has to keep its sh_addralign. The kernel has plenty + * of data whose alignment is a correctness property rather than an + * optimisation -- per-CPU variables, anything touched by an aligned SSE move, + * cacheline-aligned locks -- and a clone that lands under-aligned faults or + * silently shares a cacheline it was written to avoid. + * + * The object is new in the patched build, so klp diff has to clone it rather + * than reference the kernel's copy. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +#ifdef PATCHED +int aligned_data[2] __attribute__((aligned(64))) = { 1, 2 }; +#endif + +int target(int x) +{ +#ifdef PATCHED + return x + aligned_data[0]; +#else + return x; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/function_removal.c b/tools/objtool/tests/generic/fixtures/function_removal.c new file mode 100644 index 00000000000000..d65ff604c2c5c5 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/function_removal.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A function the patch deletes, along with its only caller's use of it. The + * original has a symbol which the patched object simply does not, so there is + * nothing to correlate it against. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +#ifndef PATCHED +int going_away(int x) +{ + return x + 7; +} +#endif + +int caller(int x) +{ +#ifdef PATCHED + return x + 1; +#else + return going_away(x); +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/init_reference.c b/tools/objtool/tests/generic/fixtures/init_reference.c new file mode 100644 index 00000000000000..9e59bdf708d3c1 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/init_reference.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Patched function referencing data in an .init section. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* volatile so the read cannot be folded into a constant, leaving no reference */ +static volatile int init_only __attribute__((section(".init.data"), used)) = 5; + +int target(int x) +{ +#ifdef PATCHED + return x + init_only + 1; +#else + return x + init_only; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/jump_label.c b/tools/objtool/tests/generic/fixtures/jump_label.c new file mode 100644 index 00000000000000..ecd3c06912af53 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/jump_label.c @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Static branch in a patched function. The jump table entry is written out by + * hand, mirroring JUMP_TABLE_ENTRY(), so the fixture builds without kernel + * headers. The key is an STT_OBJECT; anything else is ignored by + * validate_special_section_klp_reloc(). + * + * MODNAME selects whether the key is taken to belong to vmlinux or a module. + * + * NEW_KEY puts the whole static branch behind PATCHED, so the patch introduces + * one where the original had none -- a different question from patching code + * that already has a key, because the __jump_table entry itself is new. + * + * KEY_NAME renames the key. Two names are special to + * validate_special_section_klp_reloc(): a __tracepoint_* key and the + * __UNIQUE_ID_ddebug_* one pr_debug() generates are both unsupported in a + * module, but are disabled with a warning rather than rejected, because the + * kernel is full of them and refusing outright would make ordinary functions + * unpatchable. + * + * STATIC_KEY makes the key file-local. That changes the shape of the + * relocation rather than the meaning of the code: a reference to a static lands + * on the section symbol plus an addend, so the key has to be resolved from the + * section before it can be recognised as a key at all. + */ + +#ifndef MODNAME +#define MODNAME "vmlinux" +#endif + +#ifndef KEY_NAME +#define KEY_NAME klp_test_key +#endif + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=" MODNAME; + +#ifdef STATIC_KEY +static long KEY_NAME; +#else +long KEY_NAME; +#endif + +int target(int x) +{ + int r = x; + +#if defined(NEW_KEY) && !defined(PATCHED) + /* The original has no static branch at all. */ + return r + 1; +#else + asm goto( + "1: nop\n\t" + ".pushsection __jump_table, \"aw\"\n\t" + ".balign 8\n\t" + "912:\n\t" + ".pushsection .discard.annotate_data, \"M\", @progbits, 8\n\t" + ".long 912b - ., 1\n\t" + ".popsection\n\t" + ".long 1b - ., %l[l_yes] - .\n\t" + ".quad %c0 - .\n\t" + ".popsection\n\t" + : : "i" (&KEY_NAME) : : l_yes); + + r += 1; + goto out; +l_yes: + r += 2; +out: +#endif +#ifdef PATCHED + return r + 100; +#else + return r; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/klp_funcs.c b/tools/objtool/tests/generic/fixtures/klp_funcs.c new file mode 100644 index 00000000000000..3f0d3e206cef6e --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/klp_funcs.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Two changed functions and one untouched, so the patch's function list has a + * length worth checking and something that must not appear in it. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int first(int x) +{ +#ifdef PATCHED + return x + 11; +#else + return x + 1; +#endif +} + +int second(int x) +{ +#ifdef PATCHED + return x + 22; +#else + return x + 2; +#endif +} + +int third(int x) +{ + return x + 3; +} diff --git a/tools/objtool/tests/generic/fixtures/local_to_global.c b/tools/objtool/tests/generic/fixtures/local_to_global.c new file mode 100644 index 00000000000000..3c9eb9200ce54f --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/local_to_global.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A function which the patch changes from static to non-static, and a variable + * that goes the other way. The names are unchanged; only the binding moves. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* noinline, or the static one is folded into its caller and has no symbol */ +#ifdef PATCHED +__attribute__((noinline)) int flipped_up(int x) /* was static */ +#else +__attribute__((noinline)) static int flipped_up(int x) +#endif +{ + return x + 1; +} + +#ifdef PATCHED +static volatile int flipped_down = 5; /* was global */ +#else +volatile int flipped_down = 5; +#endif + +int caller(int x) +{ + flipped_down += x; +#ifdef PATCHED + return flipped_up(x) + flipped_down + 2; +#else + return flipped_up(x) + flipped_down + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/new_data.c b/tools/objtool/tests/generic/fixtures/new_data.c new file mode 100644 index 00000000000000..ac364622f12a99 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/new_data.c @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Data introduced by the patch. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +#ifdef PATCHED +/* + * Not an arithmetic progression: { 1, 2, 3, 4 } indexed by x & 3 is something + * a compiler can compute instead of load, and then target() has no reference + * to the array and there is nothing for klp diff to carry. + */ +static const int klp_new_data[4] __attribute__((used)) = { 7, 3, 11, 5 }; +#endif + +int target(int x) +{ +#ifdef PATCHED + return x + klp_new_data[x & 3]; +#else + return x; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/new_export_ref.c b/tools/objtool/tests/generic/fixtures/new_export_ref.c new file mode 100644 index 00000000000000..73210aacb4ae51 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/new_export_ref.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A reference which only exists in the patched build. The symbol has no twin + * in the original object, so what klp diff may do with it depends entirely on + * whether Module.symvers says it is exported, and by what. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +extern int newly_referenced(int x); + +/* + * A reference both builds have. When Module.symvers says a module exports + * this one, the original already depends on that module, which is what makes + * a new reference to it safe -- the loader will not let the patched module + * load without it. EXISTING_DEP leaves it out, for the case where there is + * no such dependency to inherit. + */ +extern int existing_dep(int x); + +int target(int x) +{ +#ifdef EXISTING_DEP + int base = existing_dep(x); +#else + int base = x; +#endif + +#ifdef PATCHED + return newly_referenced(base); +#else + return base + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/new_function.c b/tools/objtool/tests/generic/fixtures/new_function.c new file mode 100644 index 00000000000000..e7886eaaff3efb --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/new_function.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Function introduced by the patch. noinline keeps it from being folded. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +#ifdef PATCHED +static __attribute__((noinline)) int klp_new_helper(int x) +{ + return x * 7; +} +#endif + +int target(int x) +{ +#ifdef PATCHED + return klp_new_helper(x); +#else + return x; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/no_modinfo.c b/tools/objtool/tests/generic/fixtures/no_modinfo.c new file mode 100644 index 00000000000000..e46374702b1af7 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/no_modinfo.c @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Deliberately has no .modinfo section. */ + +int target(int x) +{ +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/special_section.c b/tools/objtool/tests/generic/fixtures/special_section.c new file mode 100644 index 00000000000000..d28c5541e33706 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/special_section.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Special section entry with no ANNOTATE_DATA_SPECIAL annotation and a local + * label at offset 0, the shape Clang produces for .kcfi_traps. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int target(int x) +{ + asm volatile( + "1:\n\t" + ".pushsection .kcfi_traps, \"a\"\n\t" + ".balign 4\n\t" + "trap_marker:\n\t" + ".long 1b - .\n\t" + ".popsection\n\t"); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/special_section_shared.c b/tools/objtool/tests/generic/fixtures/special_section_shared.c new file mode 100644 index 00000000000000..f54e24f862c60e --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/special_section_shared.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Two functions contribute to one special section; only one is patched. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int other(int x) +{ + asm volatile( + "2:\n\t" + ".pushsection .kcfi_traps, \"a\"\n\t" + ".balign 4\n\t" + ".long 2b - .\n\t" + ".popsection\n\t"); + return x * 5; +} + +int target(int x) +{ + asm volatile( + "1:\n\t" + ".pushsection .kcfi_traps, \"a\"\n\t" + ".balign 4\n\t" + ".long 1b - .\n\t" + ".popsection\n\t"); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/static_call.c b/tools/objtool/tests/generic/fixtures/static_call.c new file mode 100644 index 00000000000000..4a0c4c25321e57 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/static_call.c @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Static call site in a patched function, laid out by hand as for + * jump_label.c. MODNAME selects whether the key belongs to vmlinux or a + * module. + * + * objtool's check pass would emit the site, and klp-write-tests.txt says to + * let it. Not here: it does not emit the ANNOTATE_DATA_SPECIAL describing + * the entry boundaries -- in the kernel that comes from the static_call + * macros -- and NO_ANNOTATE below has to be able to take it away. A fixture + * which varies the annotation has to write the entry that goes with it. + * + * NO_ANNOTATE drops the ANNOTATE_DATA_SPECIAL block from the patched build, + * leaving .static_call_sites with no annotation to describe its entry + * boundaries. The section carries no entsize either, so klp diff has to fall + * back on the annotations it can still see -- and when the patched object is + * the only one that lost them, the two sides disagree about how the section is + * divided up. + * + * NEW_CALL puts the call site behind PATCHED, so the patch introduces one + * where the original had none. The .static_call_sites entry is then new, with + * nothing in the original to correlate it against. + */ + +#ifndef MODNAME +#define MODNAME "vmlinux" +#endif + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=" MODNAME; + +long __SCK__klp_test_call; + +int target(int x) +{ +#if defined(NEW_CALL) && !defined(PATCHED) + /* The original has no static call at all. */ + return x + 1; +#else + __asm__ volatile( + "1: nop\n\t" + ".pushsection .static_call_sites, \"aw\"\n\t" + ".balign 8\n\t" + "912:\n\t" +#if !(defined(PATCHED) && defined(NO_ANNOTATE)) + ".pushsection .discard.annotate_data, \"M\", @progbits, 8\n\t" + ".long 912b - ., 1\n\t" + ".popsection\n\t" +#endif + ".long 1b - ., %c0 - .\n\t" + ".popsection\n\t" + :: "i" (&__SCK__klp_test_call)); +#endif +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/static_local.c b/tools/objtool/tests/generic/fixtures/static_local.c new file mode 100644 index 00000000000000..f2f025d00a3922 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/static_local.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Static local in a patched function. */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int target(int x) +{ + static int counter; + + counter += 1; +#ifdef PATCHED + return x + counter + 1; +#else + return x + counter; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/static_local_uncorrelated.c b/tools/objtool/tests/generic/fixtures/static_local_uncorrelated.c new file mode 100644 index 00000000000000..cb4cdd7a496e3b --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/static_local_uncorrelated.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Static locals of three kinds, in one patched function. + * + * Most static locals must be correlated, so the patched code keeps using the + * running kernel's copy. Two kinds must not: + * + * - anything in .data..once, the flag behind WARN_ONCE and friends. Sharing + * it would mean a patch inherits "already warned" from before the patch. + * - the well-known names the kernel generates for such things (__warned, + * __key, __func__, ...), which are per-instance by nature. gcc names them + * . and Clang ., so both spellings have to be caught. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int target(int x) +{ + /* + * A .data..once variable whose name is *not* on the list below, so + * only the section can disqualify it. Naming it __warned would let + * the name rule catch it and the section rule go untested. + */ + static int once_flag __attribute__((section(".data..once"))); + /* a never-correlate name, in an ordinary section */ + static int __key; + /* and one that must be correlated */ + static int ordinary; + + if (!once_flag) + once_flag = 1; + __key += x; + ordinary += x; + +#ifdef PATCHED + return __key + ordinary + once_flag + 2; +#else + return __key + ordinary + once_flag + 1; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/switch_rodata.c b/tools/objtool/tests/generic/fixtures/switch_rodata.c new file mode 100644 index 00000000000000..817ddac92d817d --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/switch_rodata.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A switch dense enough that Clang builds a jump table for it, in a section of + * its own: .rodata..Lswitch.table.. + * + * The table belongs to the function and has to travel with it. It is named + * after the function but is not part of it, so klp diff has to associate the + * two rather than treating the table as unrelated data. + * + * The patch adds a case, which changes the table's contents and length. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; +const char *status_to_string(unsigned int c) +{ + switch (c) { + case 0: return "idle"; + case 1: return "running"; + case 2: return "stopped"; + case 3: return "error"; + case 4: return "paused"; + case 5: return "waiting"; + case 6: return "starting"; + case 7: return "stopping"; +#ifdef PATCHED + case 8: return "completed"; +#endif + } + return "unknown"; +} diff --git a/tools/objtool/tests/generic/fixtures/symid_discarded.c b/tools/objtool/tests/generic/fixtures/symid_discarded.c new file mode 100644 index 00000000000000..573cc2d4474b2c --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/symid_discarded.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Compiled twice and partially linked so the result has duplicate locals, + * which is what symid_needed() requires. dup_normal is in a live section, + * dup_discarded in one the vmlinux link throws away. DISCARDED_SEC selects + * which discarded section, since there is more than one and each was its own + * bug. + */ + +#ifndef DISCARDED_SEC +#define DISCARDED_SEC ".exitcall.exit" +#endif + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +static int dup_normal = 1; + +static void *dup_discarded + __attribute__((section(DISCARDED_SEC), used)) = &dup_normal; + +int FUNC_NAME(void) +{ + return dup_normal + (dup_discarded != (void *)0); +} diff --git a/tools/objtool/tests/generic/fixtures/sympos_dup.c b/tools/objtool/tests/generic/fixtures/sympos_dup.c new file mode 100644 index 00000000000000..7eded9b12cfc80 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/sympos_dup.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A static whose name recurs in every translation unit that includes it. + * Compiled once for a single-copy object and twice, partially linked, for one + * with duplicates -- which is the only case where sympos is non-zero. + * + * FUNC_NAME keeps the referencing functions distinct so both get patched. + * Only the first copy carries .modinfo; two would be a second thing to + * disambiguate and is not what this fixture is about. + */ + +#ifndef FUNC_NAME +#define FUNC_NAME use_a +#endif + +#ifndef NO_MODINFO +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; +#endif + +/* volatile so it survives as an STT_OBJECT rather than being folded away */ +static volatile int dup_counter = 1; + +int FUNC_NAME(int x) +{ + dup_counter += x; +#ifdef PATCHED + return dup_counter + 1; +#else + return dup_counter; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/sympos_vmlinux.c b/tools/objtool/tests/generic/fixtures/sympos_vmlinux.c new file mode 100644 index 00000000000000..d5e70994c582d1 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/sympos_vmlinux.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Two translation units with a same-named static, placed so that the linker + * puts them in the opposite order to the one they appear in the symbol table. + * + * VARSEC selects the section the static lands in. Linking with + * --sort-section=name then orders them alphabetically rather than by object + * order, so the first symbol in the symbol table ends up at the *higher* + * address. That is the whole point: counting symbol table order and reading + * the linked image's addresses now give different answers, which is what makes + * it possible to tell which one klp diff used. + * + * Only use_a is patched, so exactly one sympos is emitted and there is nothing + * to attribute. + */ + +#ifndef FUNC_NAME +#define FUNC_NAME use_a +#endif +#ifndef VARSEC +#define VARSEC ".data.mmm" +#endif + +#ifndef NO_MODINFO +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; +#endif + +/* volatile so it survives as an STT_OBJECT rather than being folded away */ +static volatile int dup_counter __attribute__((section(VARSEC))) = 1; + +int FUNC_NAME(int x) +{ + dup_counter += x; +#ifdef PATCHED + return dup_counter + 1; +#else + return dup_counter; +#endif +} diff --git a/tools/objtool/tests/generic/fixtures/thinlto_ambiguity.c b/tools/objtool/tests/generic/fixtures/thinlto_ambiguity.c new file mode 100644 index 00000000000000..b88830f41a9228 --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/thinlto_ambiguity.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Three translation units linked with ThinLTO, two of which have a file-local + * helper of the same name. + * + * TU_C calls into both of the others, so ThinLTO imports entry_a and entry_b + * and with them the static helper each one calls. A file-local symbol which + * has to become visible is renamed helper.llvm., and the hash is content + * derived -- so the two helpers get different hashes from each other, and + * TU_A's gets a different one again after the patch changes it. Only TU_A's + * changes: were both bodies to change, both would be cloned whichever way + * they were paired, and the pairing would not be observable. + * + * That leaves klp diff with two symbols in the original and two in the patched + * object, all four named differently, which have to be paired up correctly. + * Demangling alone gives "helper" for all of them; something else has to + * decide which is which. + * + * Only TU_A's helper changes. That is what makes a wrong pairing observable: + * paired correctly, one helper is changed and the other is not, so exactly one + * is cloned. Paired the wrong way round, both look changed -- or the wrong + * one does. If both bodies changed the outcome would be the same either way + * and the test would prove nothing. + * + * BASE differs between the two so their bodies are not identical to begin + * with. + */ + +#if defined(TU_C) +extern int entry_a(int x); +extern int entry_b(int x); +int glue(int x) { return entry_a(x) + entry_b(x + 1); } +#else +#ifdef TU_B +#define ENTRY entry_b +#define BASE 5 +#else +#define ENTRY entry_a +#define BASE 10 +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; +#endif +static __attribute__((noinline)) int helper(int x, int len) +{ + int sum = 0, i; + + for (i = 0; i < len; i++) +#if defined(PATCHED) && !defined(TU_B) + sum += i * 2 + BASE; /* only TU_A's helper changes */ +#else + sum += i + BASE; +#endif + return sum + x; +} + +int ENTRY(int x) { return helper(x, 4); } +#endif diff --git a/tools/objtool/tests/generic/fixtures/thinlto_local.c b/tools/objtool/tests/generic/fixtures/thinlto_local.c new file mode 100644 index 00000000000000..fe9f9e6bf44f9e --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/thinlto_local.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Two translation units (TU_B selects the second) linked with ThinLTO. + * Importing bump() promotes the file-local counter, renaming it + * counter.llvm.. The hash is content derived, so it differs between the + * original and patched builds. + */ + +#ifdef TU_B + +extern int bump(void); + +int other_entry(void) +{ + return bump() + bump(); +} + +#else + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +static int counter; + +int bump(void) +{ + return ++counter; +} + +int target(void) +{ +#ifdef PATCHED + return counter + 1; +#else + return counter; +#endif +} + +#endif diff --git a/tools/objtool/tests/generic/fixtures/ubsan_noise.c b/tools/objtool/tests/generic/fixtures/ubsan_noise.c new file mode 100644 index 00000000000000..bf59992541630e --- /dev/null +++ b/tools/objtool/tests/generic/fixtures/ubsan_noise.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A translation unit built with UBSAN, where only one of two functions is + * patched. + * + * Every instrumented operation gets a per-callsite metadata object in an + * anonymous data section -- .data..Lubsan_data and .data..Lubsan_type from + * GCC, .data..L__unnamed_ from Clang -- and a call to a __ubsan_handle_* + * routine. The names are compiler-generated and carry no meaning across a + * rebuild, so klp diff has to treat those sections as uncorrelated rather than + * pairing them up by name. + * + * untouched() is byte-identical in both builds and exists to catch the false + * positive: if the metadata were correlated by name, its shifts would look + * changed and it would be dragged into the patch. + * + * The shifts are what draw the instrumentation. A bounds check would do as + * well but neither compiler emits one for an index it can prove in range. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int shift_by(int v, int n); + +int untouched(int v, int n) +{ + int s = 0; + + s += v << (n & 31); + s += v << ((n + 1) & 31); + s += shift_by(v, n); + + return s; +} + +int touched(int v, int n) +{ + int s = 0; + + s += v << (n & 31); +#ifdef PATCHED + s += v << ((n + 3) & 31); +#else + s += v << ((n + 2) & 31); +#endif + + return s; +} diff --git a/tools/objtool/tests/generic/test-abs-and-addressable.sh b/tools/objtool/tests/generic/test-abs-and-addressable.sh new file mode 100755 index 00000000000000..6adb23ed4b8880 --- /dev/null +++ b/tools/objtool/tests/generic/test-abs-and-addressable.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# An absolute symbol and an __ADDRESSABLE() pointer must not disturb the +# function being patched. +# +# A SHN_ABS symbol has no section, so any walk of sym->sec which does not check +# dereferences NULL -- and the kernel has plenty, from linker scripts and from +# .set in assembly. __ADDRESSABLE() emits a pointer into .discard.addressable +# to keep a symbol referenced; it means nothing to a livepatch and is discarded +# at link time, but it is a relocation like any other and gets looked at. +# +# Neither is what the patch changes. The failure this guards against is not a +# wrong answer but a crash or an error on input the kernel produces routinely, +# which would make any function near one unpatchable. +# +# Not isolated to a single guard: the absolute symbol here has zero length, so +# it is excluded before the section check is reached and removing that check +# alone changes nothing observable. This stands as a check on the behaviour +# rather than on the line which produces it. +# +# Covers the same ground as corpus/x86_64/checksum-abs-sym-skip and +# addressable-symbols in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair abs_and_addressable.c + +# The premise: the fixture really did produce both. +in_symbols orig.o | grep -q 'ABS.*abs_sym' || + probe_skip "assembler did not make abs_sym absolute here" +assert_input_section .discard.addressable + +# Checksumming has to survive them, and still see the function that changed. +run_checksum +assert_checksum_differs target +assert_checksum_matches helper + +# So does the diff. +run_diff +assert_patched target +assert_not_patched helper + +# An absolute symbol has no address to record a checksum against, so it gets +# no entry -- the reference to it is what mattered, not the symbol itself. +in_relocs orig.o | awk '/rela\.discard\.sym_checksum/,/^$/' | grep -qw abs_sym && + fail "absolute symbol got a checksum entry" + +pass "absolute and __ADDRESSABLE symbols do not disturb the patched function" diff --git a/tools/objtool/tests/generic/test-basic.sh b/tools/objtool/tests/generic/test-basic.sh new file mode 100755 index 00000000000000..562edfbc8646f6 --- /dev/null +++ b/tools/objtool/tests/generic/test-basic.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Only functions whose code changed get cloned into the patch. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair basic.c +run_diff + +assert_patched changed +assert_not_patched untouched +assert_section ".init.klp_funcs" +assert_section ".init.klp_objects" + +pass "changed function cloned, unchanged function left alone" diff --git a/tools/objtool/tests/generic/test-changed-data.sh b/tools/objtool/tests/generic/test-changed-data.sh new file mode 100755 index 00000000000000..c5c7381bb4091b --- /dev/null +++ b/tools/objtool/tests/generic/test-changed-data.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Livepatching replaces functions, not data. A changed data symbol must be +# rejected. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair changed_data.c +run_diff 255 + +diff_log | grep -q 'changed data: klp_test_data' || + fail "expected rejection, got: $(diff_log | tail -1)" +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "changed data symbol rejected" diff --git a/tools/objtool/tests/generic/test-checksum-data.sh b/tools/objtool/tests/generic/test-checksum-data.sh new file mode 100755 index 00000000000000..e915026b79a73f --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-data.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# What a data object's checksum has to cover. +# +# checksum_update_object() hashes the symbol's length, its bytes (when the +# section has any -- .bss does not), and then +# every relocation it carries -- as the target's name plus the adjusted addend, +# except for a reference into a string section, which contributes the string's +# contents instead. +# +# Each of those is load-bearing, and the failure is always the same shape: a +# checksum that ignores one of them calls a changed object unchanged, klp diff +# leaves it out of the patch, and the patched code goes on reading the +# kernel's old copy. Nothing says so at build time. +# +# The string case is the one that cannot be caught by hashing bytes alone. The +# pointer is identical -- same section, same offset -- and only the text it +# refers to moved. +# +# Covers the same ground as corpus/x86_64/checksum-data-basic, +# checksum-data-func-ptr, checksum-data-string-ptr and checksum-string-reloc in +# Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup + +# check +# +# Build the pair with one difference and require that symbol's checksum to move. +check() +{ + build_pair checksum_data.c "-D$1" + run_checksum + + assert_checksum_differs "$2" +} + +# The object's own bytes. +check PLAIN_VALUE plain +# Its length, for a .bss object whose bytes are not hashed at all. +check LONGER sized +# A relocation's target: same bytes in the object, different symbol named. +check WHICH_FUNC descriptor +check WHICH_STR descriptor +# The contents of a string the object points at, with the pointer untouched. +check STR_CONTENT descriptor +# A relocation's addend: same target symbol, different offset into it. +check WHICH_SLOT descriptor +# The same, for a static reached through its section symbol: the reference has +# to be resolved back to the object before there is a name or offset to hash. +check WHICH_PRIV descriptor + +# Having shown five things that must change it, show one that must not: an +# unrelated edit elsewhere in the file leaves this object alone. +build_pair checksum_data.c -DPLAIN_VALUE +run_checksum +assert_checksum_matches descriptor + +pass "data checksums cover length, bytes, reloc targets and string contents" diff --git a/tools/objtool/tests/generic/test-checksum-debug.sh b/tools/objtool/tests/generic/test-checksum-debug.sh new file mode 100755 index 00000000000000..78856b1916362a --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-debug.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# "klp checksum --debug-checksum" prints a per-instruction checksum stream, +# and klp-build -f (--show-first-changed) parses it to report where a function +# first differs between the original and patched builds. +# +# It is a debugging aid, so nothing fails when it breaks: klp-build greps the +# stream, and an unmatched grep just yields no output, which reads as "no +# instruction changed". That is exactly how the format drifted out from under +# it once already. Pin the shape klp-build depends on: +# +# DEBUG: : checksum: (): +0x <16 hex digits> +# +# and that --dry-run leaves the object alone, since klp-build runs this against +# objects it is going to checksum again for real. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair basic.c + +before="$(md5sum < "$workdir/orig.o")" + +"$OBJTOOL" klp checksum --dry-run --debug-checksum=changed \ + "$workdir/orig.o" > "$workdir/debug.log" 2>&1 || + fail "klp checksum --debug-checksum failed" + +# --dry-run has to mean it: klp-build checksums these objects again afterwards, +# and "already has .discard.sym_checksum, skipping" would lose the real run. +[ "$(md5sum < "$workdir/orig.o")" = "$before" ] || + fail "--dry-run modified the object" +has_input_section orig.o .discard.sym_checksum && + fail "--dry-run created .discard.sym_checksum" + +grep -qE '^DEBUG: .*: checksum: changed\(\): [^ ]+\+0x[0-9a-f]+ [0-9a-f]{16}$' \ + "$workdir/debug.log" || + fail "unexpected --debug-checksum format: $(head -1 "$workdir/debug.log")" + +# This is the pattern klp-build greps with. Keep it working verbatim. +grep -qE "^DEBUG: .*checksum: changed\(\): " "$workdir/debug.log" || + fail "klp-build's --show-first-changed pattern no longer matches" + +# Only the requested function, or klp-build attributes instructions to the +# wrong one. +grep -qE 'checksum: untouched\(\)' "$workdir/debug.log" && + fail "--debug-checksum=changed also dumped untouched()" + +pass "--debug-checksum format is the one klp-build -f parses" diff --git a/tools/objtool/tests/generic/test-checksum-insn.sh b/tools/objtool/tests/generic/test-checksum-insn.sh new file mode 100755 index 00000000000000..e1c04a1f518a4b --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-insn.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# What a function's checksum has to cover beyond its instruction bytes. +# +# checksum_update_insn() hashes the raw bytes, and then what any relocation on +# the instruction refers to: a string section contributes the string's +# contents, anything else the target symbol's name and the adjusted addend, +# with a reference to a static resolved back through its section symbol first. +# +# None of these show up in the bytes. A rel32 operand is zero in the object +# and supplied by the relocation, so every change below leaves the encoded +# instruction byte-identical. A checksum stopping at the bytes reports the +# function unchanged, klp diff omits it, and the patch silently does not +# contain the fix. +# +# test-checksum-position is the other half of this: it covers what must *not* +# change the checksum when a function merely moves. +# +# Covers the same ground as corpus/x86_64/checksum-reloc-sym, +# checksum-pc-relative-addend, checksum-string-reloc and +# checksum-sec-sym-resolve in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup + +# check +check() +{ + build_pair checksum_insn.c "-D$1" + run_checksum + + # The premise for all of them: the operand is a relocation, not bytes. + assert_checksum_differs target +} + +check WHICH_CALL # relocation target name +check STR_CONTENT # contents of a string the code passes +check WHICH_SLOT # addend, same target symbol +check WHICH_PRIV # addend via a static's section symbol + +# The converse: rebuilding identical source leaves it alone, so the above is +# not just "any rebuild moves the checksum". +build_pair checksum_insn.c +run_checksum +assert_checksum_matches target + +pass "instruction checksums cover reloc targets, addends and string contents" diff --git a/tools/objtool/tests/generic/test-checksum-position.sh b/tools/objtool/tests/generic/test-checksum-position.sh new file mode 100755 index 00000000000000..5ae759e719eb80 --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-position.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A function's checksum must not depend on where the function sits. +# +# A jump or call without a relocation encodes its target as an offset from the +# instruction. Hashing those bytes makes the checksum change whenever anything +# ahead of the function changes size -- so an unrelated edit elsewhere in the +# file reports this function as changed too, and the patch grows to include it +# and everything it references. Nothing fails; the livepatch is just larger and +# riskier than the patch it came from. +# +# Here the "patch" moves target() away from callee() and changes nothing else: +# both are aligned to 64 in the patched build, which shifts them apart without +# touching a byte of either. See the fixture for why it is done that way. + +. "$(dirname "$0")/../lib.sh" + +setup + +# -fno-function-sections, or each function is at offset 0 of its own section +# and target() never moves. +build_pair checksum_position.c -fno-function-sections + +assert_input_symbol target + +# The fixture is only meaningful if the displacement target's calls encode +# actually changed, and that is the distance to callee() -- not target's own +# offset. A compiler which shifted the two by the same amount would move +# target and leave the distance alone, and then the bytes are identical and +# the checksum matches for the uninteresting reason. Ask about the distance. +sym_off() # $1 object, $2 symbol +{ + in_symbols "$1" | awk -v n="$2" '$NF == n { print $2; exit }' +} + +orig_t="$(sym_off orig.o target)"; orig_c="$(sym_off orig.o callee)" +new_t="$(sym_off patched.o target)"; new_c="$(sym_off patched.o callee)" + +[ -n "$orig_t" ] && [ -n "$orig_c" ] && [ -n "$new_t" ] && [ -n "$new_c" ] || + fail "target or callee missing from one of the objects" + +orig_gap=$(( 16#$orig_t - 16#$orig_c )) +new_gap=$(( 16#$new_t - 16#$new_c )) +[ "$orig_gap" != "$new_gap" ] || + probe_skip "this compiler kept target() and callee() the same distance" \ + "apart; the call displacement did not change" + +assert_checksum_matches target + +pass "checksum unchanged when the function only moves" diff --git a/tools/objtool/tests/generic/test-checksum-skip.sh b/tools/objtool/tests/generic/test-checksum-skip.sh new file mode 100755 index 00000000000000..f245535a0f363f --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-skip.sh @@ -0,0 +1,81 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Symbols which must not get a checksum entry of their own. +# +# .discard.sym_checksum is an array of { address, checksum } looked up by the +# address a relocation points at, so the invariant is one entry per address. +# calculate_checksums() skips three kinds of symbol to keep it: +# +# zero-length nothing to hash, and its address belongs to whatever really +# lives there +# alias a second name for an address already covered +# cold part hashed into its parent, which func_for_each_insn() walks +# into, so its own entry would double-count +# +# A duplicate entry is not a build failure. It makes the lookup ambiguous, and +# whichever checksum loses is simply never consulted again -- so a function +# whose code changed can be read as unchanged and dropped from the patch. +# +# Of the three, only the alias skip is isolated here: removing it makes this +# test fail. A zero-length symbol is excluded by more than one of the guards +# at once -- its section has no data either -- so no single change makes that +# assertion fail, and it stands as a check on the behaviour rather than on the +# line which produces it. Nothing here reaches the cold-part skip, which +# wants a compiler that splits functions; test-cold-function covers that +# symbol surviving into the patch, not its checksum. +# +# Covers the same ground as corpus/x86_64/checksum-zero-len-sym, +# checksum-alias-skip and checksum-cold-skip in Joe Lawrence's klp-build unit +# test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair checksum_skip.c + +assert_input_symbol empty_marker +assert_input_symbol alias_function +run_checksum + +# entries_for [-t] +# +# The symbol names .discard.sym_checksum has an entry for, one per line. With +# -t, what each entry points at instead: the name and its addend, which is the +# address the kernel looks the entry up by. A name alone is not that address, +# since a relocation against a section symbol names the section and puts the +# offset in the addend, and several entries can then share a name honestly. +entries_for() +{ + in_relocs "$1" | awk -v target="${2:-}" '/rela\.discard\.sym_checksum/,/^$/ { + if ($1 !~ /^[0-9a-f]{8,}/) + next + if (target == "-t") + print $5, $6, $7 + else + print $5 + }' +} + +entries="$(entries_for orig.o)" + +# The control: something real did get an entry, so an empty listing cannot +# make the rest of this pass by default. +echo "$entries" | grep -qx target || + fail "no checksum entry for target" + +echo "$entries" | grep -qx empty_marker && + fail "zero-length symbol got a checksum entry" + +# One of the two names for that address is kept and the other skipped; which +# one falls out of symbol table order and is not the point. Two would be. +n="$(echo "$entries" | grep -cxE 'real_function|alias_function')" +[ "$n" = 1 ] || + fail "expected 1 checksum entry across real_function and its alias, found $n" + +# One entry per address, which is what the skipping is for. +dupes="$(entries_for orig.o -t | sort | uniq -d)" +[ -z "$dupes" ] || + fail "two checksum entries for one address: $dupes" + +pass "zero-length symbols and aliases get no checksum entry of their own" diff --git a/tools/objtool/tests/generic/test-checksum-value.sh b/tools/objtool/tests/generic/test-checksum-value.sh new file mode 100755 index 00000000000000..feae6a12e98d8d --- /dev/null +++ b/tools/objtool/tests/generic/test-checksum-value.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# The per-function checksums klp checksum records are what klp diff uses to +# decide which functions changed. A checksum covering too little misses a real +# change and the patch silently omits the function; one covering too much, or +# unstable across identical input, clones functions nobody patched and drags +# their dependencies in with them. +# +# test-basic covers which functions got cloned, which is downstream of this and +# passes for either kind of wrong checksum as long as the two errors do not +# happen to cancel. This checks the checksums themselves. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair basic.c + +assert_input_symbol changed +assert_input_symbol untouched + +run_checksum + +# The edited function's checksum has to move, the untouched one's must not. +assert_checksum_differs changed +assert_checksum_matches untouched + +# And it has to be a function of the code, not of the build: checksumming the +# same input twice has to give the same answer, or every rebuild reports +# spurious changes. +first="$(checksum_of orig.o changed)" +build_pair basic.c +run_checksum +[ "$(checksum_of orig.o changed)" = "$first" ] || + fail "checksum for 'changed' differs between builds of identical source" + +pass "checksums track the changed function and are stable across rebuilds" diff --git a/tools/objtool/tests/generic/test-cold-function.sh b/tools/objtool/tests/generic/test-cold-function.sh new file mode 100755 index 00000000000000..a02652fe534139 --- /dev/null +++ b/tools/objtool/tests/generic/test-cold-function.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Both halves of a split function belong to the patch; carrying only the hot +# part leaves the cold path branching into unpatched code. + +. "$(dirname "$0")/../lib.sh" + +# Clang does not split functions into a cold part at all, so there is nothing +# for this test to look at there. A given gcc may or may not split, which is a +# version property rather than a compiler choice -- that stays a probe below. +gcc_only "clang does not split functions into a cold part" + +setup + +split_flag=-freorder-blocks-and-partition +cc_supports "$split_flag" || split_flag= + +build_pair cold_function.c $split_flag + +# Find what the compiler called the cold half -- target.cold, target.cold.0, +# depending on version -- and name it exactly from here on. +cold_sym="$(in_symbols orig.o | awk '$NF ~ /^target\.cold/ { print $NF; exit }')" +[ -n "$cold_sym" ] || + probe_skip "compiler did not split the function into a cold part" + +run_diff + +assert_patched target +# Match the name field exactly, and require it to be defined. Had the cold +# half been left behind, the branch to it would appear as an undefined +# .klp.sym.vmlinux.target.cold,0 -- a different name, which happens to contain +# this one. Asking about a column instead of the name would not tell them +# apart: readelf prints SHN_LIVEPATCH as "OS [0xff20]" and llvm-readelf as +# "OS[0xff20]", so the fields either side of the name shift between the two. +out_symbols | awk -v n="$cold_sym" '$NF == n && $(NF - 1) != "UND"' | grep -q . || + fail "cold half ($cold_sym) was not carried into the patch" + +pass "cold half carried into the patch with its parent" diff --git a/tools/objtool/tests/generic/test-data-alignment.sh b/tools/objtool/tests/generic/test-data-alignment.sh new file mode 100755 index 00000000000000..8e389544a3b16c --- /dev/null +++ b/tools/objtool/tests/generic/test-data-alignment.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A cloned data section keeps its alignment. +# +# Plenty of kernel data is aligned for correctness rather than speed: per-CPU +# variables, anything touched by an aligned vector move, structures padded to +# own a cacheline. A clone that lands under-aligned either faults on first use +# or silently shares a line it was laid out to avoid, and neither shows up +# until the patch is loaded on hardware that cares. +# +# Fixed by 2f2600decb30 ("objtool/klp: Fix alignment of cloned data +# sections"). +# +# Covers the same ground as corpus/x86_64/cloned-data-alignment in Joe +# Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair data_alignment.c + +# The premise: the compiler really did over-align it, and the object is new in +# the patch so it has to be cloned rather than referenced. +want="$(in_sections patched.o | sed 's/^ *\[[ 0-9]*\] *//' | + awk '$1 == ".data.aligned_data" { print $NF }')" +[ "$want" = 64 ] || + probe_skip "compiler gave .data.aligned_data alignment '$want', not 64" +has_input_section orig.o .data.aligned_data && + fail "fixture put aligned_data in the original; nothing to clone" + +run_diff +assert_section .data.aligned_data + +got="$(out_sections | sed 's/^ *\[[ 0-9]*\] *//' | + awk '$1 == ".data.aligned_data" { print $NF }')" +[ "$got" = "$want" ] || + fail "cloned .data.aligned_data has alignment $got, expected $want" + +pass "cloned data section keeps its alignment" diff --git a/tools/objtool/tests/generic/test-export-symbol-for-modules.sh b/tools/objtool/tests/generic/test-export-symbol-for-modules.sh new file mode 100755 index 00000000000000..7e7bdde6a7aca7 --- /dev/null +++ b/tools/objtool/tests/generic/test-export-symbol-for-modules.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# EXPORT_SYMBOL_FOR_MODULES() puts a vmlinux symbol in a "module:" +# namespace, and the module loader grants access by matching the importing +# module's name against that list. A livepatch module is never on the list, so +# referencing such a symbol with a normal relocation fails modpost, and if that +# is silenced, fails to load with "Unknown symbol". It needs a klp relocation, +# the same as an unexported symbol. +# +# Ordinary namespaces are not affected: copy_import_ns() propagates the patched +# object's import tags to the patch module, so a normal relocation works. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair cross_module.c + +sym=other_mod_func + +# Plain vmlinux export: a normal relocation is what we want. +export_syms "$sym" +run_diff +assert_no_klp_sym "$sym" + +# Ordinary namespace: still a normal relocation. +export_syms +add_exports_ns vmlinux MY_NS "$sym" +run_diff +assert_no_klp_sym "$sym" + +# module: namespace: has to become a klp relocation. +export_syms +add_exports_ns vmlinux module:kvm "$sym" +run_diff +assert_klp_sym "$sym" vmlinux +assert_section __klp_relocs.vmlinux + +pass "EXPORT_SYMBOL_FOR_MODULES symbol referenced with a klp relocation" diff --git a/tools/objtool/tests/generic/test-function-removal.sh b/tools/objtool/tests/generic/test-function-removal.sh new file mode 100755 index 00000000000000..df76e81e3936c4 --- /dev/null +++ b/tools/objtool/tests/generic/test-function-removal.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A patch which deletes a function leaves a symbol in the original with no +# counterpart in the patched object. klp diff cannot correlate it, and must +# say so and carry on: livepatching cannot remove code from a running kernel, +# so what matters is that the surviving caller is patched and the deleted +# function is not dragged into the patch module. +# +# Cloning it would be worse than useless -- dead code in the patch, plus +# whatever it references, resolved against a kernel where it may not exist. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair function_removal.c + +# One-sided by construction: present in the original, gone from the patched. +has_input_symbol orig.o going_away || + fail "fixture has no going_away in the original" +has_input_symbol patched.o going_away && + fail "fixture still has going_away in the patched object" + +run_diff + +assert_diff_log 'no correlation: going_away' + +# The caller changed, so it is patched ... +assert_patched caller +# ... and the deleted function comes along in no form at all. +assert_not_patched going_away +assert_no_symbol going_away + +pass "deleted function reported as uncorrelated and left out of the patch" diff --git a/tools/objtool/tests/generic/test-init-reference.sh b/tools/objtool/tests/generic/test-init-reference.sh new file mode 100755 index 00000000000000..921cf493cd7909 --- /dev/null +++ b/tools/objtool/tests/generic/test-init-reference.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Init code and data are freed after boot, so a klp relocation against them can +# never resolve. Such a patch must be rejected. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair init_reference.c + +# The rejection can only happen if the patched build really does reference the +# init symbol. The fixture makes init_only volatile so the read cannot be +# folded away, and this checks that it worked: without a relocation klp diff +# would succeed, and the failure below would read as a missing check rather +# than as a fixture which stopped posing the question. +# Either spelling will do. A file-local variable is reached through its +# section symbol -- .init.data -- and a global one by name; which of the two +# the compiler picks is its business, and the reference is what matters. +in_relocs "$patched_obj" | + awk '$5 == ".init.data" || $5 == "init_only"' | grep -q . || + fail "patched object has no reference into .init.data; the fixture tests nothing" + +run_diff 255 + +diff_log | grep -q "can't patch or reference init code/data" || + fail "expected rejection, got: $(diff_log | tail -1)" + +pass "reference to init data rejected" diff --git a/tools/objtool/tests/generic/test-jump-label-exempt-keys.sh b/tools/objtool/tests/generic/test-jump-label-exempt-keys.sh new file mode 100755 index 00000000000000..fa2f913d860b36 --- /dev/null +++ b/tools/objtool/tests/generic/test-jump-label-exempt-keys.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Two kinds of module-owned static branch key are disabled with a warning +# instead of rejected. +# +# A module-local key is normally fatal: late module patching lets the livepatch +# load before the module it depends on, so the unresolved __jump_table entry is +# dereferenced by jump_label_add_module(). test-jump-label-module-key covers +# that rejection. +# +# Tracepoints and pr_debug() generate such keys everywhere, though, and +# refusing them outright would make any function containing a trace_*() call or +# a pr_debug() unpatchable. So klp diff drops the entry, says so, and carries +# on: the patched code keeps working with that one tracepoint or debug print +# permanently off. +# +# Both halves matter. A build that fails is a function nobody can patch; an +# entry left in place is the memory corruption the rejection exists to prevent. +# +# The two exemptions are isolated: remove either and this fails. That the +# entry is then dropped is asserted but not isolated -- making the caller keep +# it anyway produces no output difference here, so that assertion stands as a +# check on the behaviour rather than on the line which produces it. +# +# Covers the same ground as corpus/x86_64/static-call-module-tracepoint and +# pr-debug-unsupported in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup + +# check +check() +{ + build_pair jump_label.c -DKEY_NAME="$1" -DMODNAME='"klp_testmod"' + require_input_section __jump_table + + # Accepted, not rejected: this is the whole point. + run_diff + assert_diff_log "$2" + + # And the entry is gone, not merely complained about. + assert_patched target + assert_no_section __jump_table +} + +check __tracepoint_klp_test 'disabling unsupported tracepoint klp_test' +check __UNIQUE_ID_ddebug_klp_test 'disabling unsupported pr_debug' + +pass "tracepoint and pr_debug keys disabled with a warning, not rejected" diff --git a/tools/objtool/tests/generic/test-jump-label-key.sh b/tools/objtool/tests/generic/test-jump-label-key.sh new file mode 100755 index 00000000000000..142f94bd38ecf2 --- /dev/null +++ b/tools/objtool/tests/generic/test-jump-label-key.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A cloned __jump_table entry must keep a relocation in its key slot, whether +# the key needs a klp relocation or not. + +. "$(dirname "$0")/../lib.sh" + +key=klp_test_key + +setup +build_pair jump_label.c + +has_input_section orig.o __jump_table || + probe_skip "fixture produced no __jump_table on this arch" + +key_slot_relocs() +{ + out_relocs | awk '/rela__jump_table/,/^$/' | grep -c "^0*8[[:space:]]" +} + +# Unexported: klp relocation, key slot holds a tombstone. +export_syms +run_diff + +[ "$(key_slot_relocs)" = 1 ] || + fail "unexported key: key slot has no relocation" +out_relocs | awk '/rela__jump_table/,/^$/' | grep -q "\.klp\.tombstone\.$key" || + fail "unexported key: expected a .klp.tombstone.$key relocation" +out_symbols | grep -q "\.klp\.sym\..*\.$key," || + fail "unexported key: no .klp.sym reference for the real relocation" + +# Exported: ordinary relocation, no klp machinery. +export_syms "$key" +run_diff + +[ "$(key_slot_relocs)" = 1 ] || + fail "exported key: key slot has no relocation" +out_relocs | awk '/rela__jump_table/,/^$/' | grep -q "[[:space:]]$key[[:space:]]*+" || + fail "exported key: expected a direct relocation to $key" +out_symbols | grep -q '\.klp\.tombstone\.' && + fail "exported key: tombstone emitted for an exported symbol" + +pass "key slot populated for exported and unexported vmlinux keys" diff --git a/tools/objtool/tests/generic/test-jump-label-module-key.sh b/tools/objtool/tests/generic/test-jump-label-module-key.sh new file mode 100755 index 00000000000000..7c3f82cdd3c1a1 --- /dev/null +++ b/tools/objtool/tests/generic/test-jump-label-module-key.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A static branch key owned by a module cannot be reached with a klp +# relocation and must be rejected. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair jump_label.c -DMODNAME='"klp_testmod"' + +has_input_section orig.o __jump_table || + probe_skip "fixture produced no __jump_table on this arch" + +run_diff 255 + +diff_log | grep -q 'unsupported static branch key klp_test_key' || + fail "expected rejection, got: $(diff_log | tail -1)" +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "module-owned static branch key rejected" diff --git a/tools/objtool/tests/generic/test-jump-label-module-static-key.sh b/tools/objtool/tests/generic/test-jump-label-module-static-key.sh new file mode 100755 index 00000000000000..7bdcbaf2fa60ad --- /dev/null +++ b/tools/objtool/tests/generic/test-jump-label-module-static-key.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A static branch key owned by a module is rejected whether the key is global +# or file-local. +# +# The rejection matters because late module patching allows the livepatch +# module to load before the module it depends on: the __jump_table klp reloc is +# then unresolved, and jump_label_add_module() dereferences an uninitialized +# pointer. Catching it at build time is the only defence. +# +# test-jump-label-module-key covers the global key. A file-local one reaches +# the same check by a different route: the compiler emits the reference against +# the section symbol plus an addend, so validate_special_section_klp_reloc() +# has to resolve it to the underlying object before it can see a key at all. +# Until it did, a static key was passed over as "not STT_OBJECT" and the +# unsupported reference was emitted with nothing said. +# +# Fixed by f9fb44b0ecef ("objtool/klp: Fix detection of corrupt static +# branch/call entries"). + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair jump_label.c -DSTATIC_KEY -DMODNAME='"klp_testmod"' + +require_input_section __jump_table + +# The premise: the key is reached through its section symbol, not by name. +# Without that this is just a second copy of test-jump-label-module-key. +input_jump_relocs="$(in_relocs orig.o | awk '/rela__jump_table/,/^$/')" + +echo "$input_jump_relocs" | grep -q klp_test_key || + fail "fixture produced no __jump_table reference to the key" +echo "$input_jump_relocs" | grep -qE '\.(bss|data)\.klp_test_key' || + probe_skip "compiler referenced the static key by name, not through its section" + +run_diff 255 + +diff_log | grep -q 'unsupported static branch key klp_test_key' || + fail "expected rejection, got: $(diff_log | tail -1)" +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "module-owned file-local static branch key rejected" diff --git a/tools/objtool/tests/generic/test-jump-label-new-key.sh b/tools/objtool/tests/generic/test-jump-label-new-key.sh new file mode 100755 index 00000000000000..3bc6005fc776c8 --- /dev/null +++ b/tools/objtool/tests/generic/test-jump-label-new-key.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A patch may introduce a static branch where the original function had none. +# +# That is not the same case as patching a function which already has one. The +# __jump_table entry is itself new, so there is no counterpart in the original +# to correlate it against: klp diff has to carry the entry and the key into the +# patch from scratch, and the key has to be reached the way any other reference +# to a vmlinux symbol is. +# +# Get it wrong and the entry is dropped, leaving a static branch the kernel +# never patches -- the code takes the wrong arm forever, silently. +# +# Where the key lives still decides whether that is allowed, exactly as it does +# for a key the original already had: a module-owned one cannot be reached, so +# introducing one has to stop the build rather than emit an entry nothing will +# resolve. +# +# Covers the same ground as corpus/x86_64/static-branch-vmlinux-new and +# static-branch-module-new in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup klp_test_key +build_pair jump_label.c -DNEW_KEY + +# The premise: the original really has no jump table, and the patched one does. +has_input_section orig.o __jump_table && + fail "fixture put a __jump_table in the original; nothing new to add" +has_input_section patched.o __jump_table || + probe_skip "compiler produced no __jump_table on this arch" + +run_diff + +assert_patched target +assert_section __jump_table +assert_reloc_sym __jump_table target + +# The same new branch, with the key owned by a module. Drop the vmlinux export +# first: while it is exported the key is reachable and being new changes +# nothing, which is what the first version of this got wrong. +export_syms +rm -f "$workdir/out.o" +build_pair jump_label.c -DNEW_KEY -DMODNAME='"klp_testmod"' +run_diff 255 +assert_diff_log 'unsupported static branch key klp_test_key' +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "static branch introduced by the patch carried in, or rejected for a module key" diff --git a/tools/objtool/tests/generic/test-klp-funcs-content.sh b/tools/objtool/tests/generic/test-klp-funcs-content.sh new file mode 100755 index 00000000000000..32fbd5f6a6fa91 --- /dev/null +++ b/tools/objtool/tests/generic/test-klp-funcs-content.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# .init.klp_funcs is the list the kernel walks to decide what to patch, and +# .init.klp_objects points at it. Existing tests assert only that the sections +# exist, which they do whether the list names the right functions, the wrong +# ones, or none at all -- and a patch module with an empty function list loads +# perfectly happily and patches nothing. +# +# Each entry pairs a name string in .rodata.klp.str1.1 with a relocation to the +# new function, so both halves are checkable. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair klp_funcs.c +run_diff + +assert_section .init.klp_funcs +assert_section .init.klp_objects + +# Two functions changed, so two entries, each contributing a name relocation +# and a function relocation. +assert_reloc_count .init.klp_funcs 4 + +# The functions that changed are named ... +assert_reloc_sym .init.klp_funcs first +assert_reloc_sym .init.klp_funcs second +# ... and the one that did not is absent, from the list and from the patch. +assert_no_reloc_sym .init.klp_funcs third +assert_not_patched third + +# The names the kernel matches on are real strings, not just relocations. +# readelf prints one per line as "[ offset] ", so compare the whole +# name: a word-boundary match would also accept ".text.first", since a dot is +# not a word character. +for name in first second; do + out_strings .rodata.klp.str1.1 | awk -v n="$name" '$NF == n' | grep -q . || + fail "no '$name' string in .rodata.klp.str1.1" +done + +# The object list has to reach the function list, or nothing is walked. +assert_reloc_sym .init.klp_objects .init.klp_funcs + +pass "klp_funcs lists exactly the changed functions, by name and relocation" diff --git a/tools/objtool/tests/generic/test-local-to-global-flip.sh b/tools/objtool/tests/generic/test-local-to-global-flip.sh new file mode 100755 index 00000000000000..45e92797190b61 --- /dev/null +++ b/tools/objtool/tests/generic/test-local-to-global-flip.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A patch can change a symbol's linkage without renaming it: dropping "static" +# from a helper so something else can call it, or adding it to one that is no +# longer shared. Correlation keys off more than the name, so a symbol whose +# binding moved can fail to pair with itself. +# +# Failing to correlate is not a build failure. The symbol looks new, and a +# "new" data symbol is either rejected or cloned as a second copy -- at which +# point the patched code updates its own private variable and the rest of the +# kernel keeps reading the original. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair local_to_global.c + +# Confirm the fixture really moved the bindings, in both directions. +# +# Match the binding and the name as fields, not as substrings of the line. +# -ffunction-sections and -fdata-sections give these symbols sections of their +# own, and the section symbols -- .text.flipped_up, .data.flipped_down -- are +# always LOCAL, so "does a LOCAL line mention flipped_up" is answered by the +# wrong symbol. GNU readelf 2.35 happens to leave section symbol names blank, +# but llvm-readelf prints them, and a premise that holds on one readelf and +# not the other is no premise at all. +has_binding() # $1 object, $2 binding, $3 symbol +{ + in_symbols "$1" | awk -v b="$2" -v n="$3" '$5 == b && $NF == n' | grep -q . +} + +has_binding orig.o LOCAL flipped_up || + fail "flipped_up is not local in the original" +has_binding patched.o GLOBAL flipped_up || + fail "flipped_up is not global in the patched object" +has_binding orig.o GLOBAL flipped_down || + fail "flipped_down is not global in the original" +has_binding patched.o LOCAL flipped_down || + fail "flipped_down is not local in the patched object" + +run_diff + +assert_diff_log 'changed function: caller' + +# Correlated means each pairs with its own counterpart in the original, so the +# patch refers back to the kernel's copy ... +assert_klp_sym flipped_up vmlinux +assert_klp_sym flipped_down vmlinux + +# ... rather than carrying its own. A second copy of flipped_down is the bad +# outcome: patched code would update its private one while the rest of the +# kernel keeps reading the original. +assert_not_patched flipped_up +assert_no_section .data.flipped_down +assert_no_section .bss.flipped_down + +diff_log | grep -q 'no correlation' && + fail "linkage change reported as an uncorrelated symbol" +diff_log | grep -q 'changed data' && + fail "linkage change reported as changed data" + +pass "symbols correlated across a change of linkage" diff --git a/tools/objtool/tests/generic/test-local-vs-export.sh b/tools/objtool/tests/generic/test-local-vs-export.sh new file mode 100755 index 00000000000000..5b91101dadd64b --- /dev/null +++ b/tools/objtool/tests/generic/test-local-vs-export.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# find_export() matched on symbol name alone, so a static function or variable +# sharing a name with an export was mistaken for a reference to that export. +# For a vmlinux export that means no klp relocation at all: the normal +# relocation left behind is resolved by the module loader to the vmlinux +# symbol, and the patched code quietly reads and writes the wrong object. +# +# Exports are always global, so a local symbol is never one. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_local.c + +# The static the fixture uses. Compilers mangle statics variously -- gcc says +# counter.0, clang says target.counter -- so find what this one produced rather +# than assuming a shape. +local_sym="$(in_symbols orig.o | + awk '$4 == "OBJECT" && $5 == "LOCAL" && $8 ~ /counter/ { print $8; exit }')" +[ -n "$local_sym" ] || + fail "fixture produced no local 'counter' symbol" + +# Contrive the collision: something else exports that same name. +export_syms "$local_sym" counter +run_diff + +# Still treated as the local it is, not as the export. +assert_klp_sym "$local_sym" vmlinux + +pass "local symbol not mistaken for an export of the same name" diff --git a/tools/objtool/tests/generic/test-missing-checksum.sh b/tools/objtool/tests/generic/test-missing-checksum.sh new file mode 100755 index 00000000000000..89a05b1869edbf --- /dev/null +++ b/tools/objtool/tests/generic/test-missing-checksum.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Without .discard.sym_checksum there is nothing to compare; concluding that +# nothing changed would be worse than failing. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair basic.c + +( cd "$workdir" && "$OBJTOOL" klp diff orig.o patched.o out.o ) \ + > "$workdir/diff.log" 2>&1 && fail "klp diff accepted an unchecksummed object" + +grep -q 'sym_checksum' "$workdir/diff.log" || + fail "expected a complaint about the checksum section, got: $(tail -1 "$workdir/diff.log")" + +pass "unchecksummed input rejected" diff --git a/tools/objtool/tests/generic/test-missing-modinfo.sh b/tools/objtool/tests/generic/test-missing-modinfo.sh new file mode 100755 index 00000000000000..6f242f7eb756e3 --- /dev/null +++ b/tools/objtool/tests/generic/test-missing-modinfo.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# The module name in .modinfo ends up in the livepatch's klp_object, so an +# object without one cannot be diffed. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair no_modinfo.c +run_diff 255 + +diff_log | grep -q 'modinfo' || + fail "expected a complaint about .modinfo, got: $(diff_log | tail -1)" + +pass "object without .modinfo rejected" diff --git a/tools/objtool/tests/generic/test-modname-normalize.sh b/tools/objtool/tests/generic/test-modname-normalize.sh new file mode 100755 index 00000000000000..a8059bdbd66705 --- /dev/null +++ b/tools/objtool/tests/generic/test-modname-normalize.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Module.symvers records the build-tree path of the object that exports a +# symbol, not the name the module has at runtime: "arch/x86/kvm/kvm-intel", +# where the kernel knows the module as "kvm_intel". +# +# The klp symbol name embeds the owning object, and livepatch matches it +# against loaded modules by name. Left unnormalized it names a module that +# does not exist, and the relocation is never resolved -- at load time, with no +# build-time complaint. + +. "$(dirname "$0")/../lib.sh" + +setup +build_module_pair cross_module.c klp_testmod + +# A path with directory components, a dash, and no extension. +export_syms +add_exports "arch/x86/kvm/kvm-intel" other_mod_func +run_diff + +# Directories stripped, dash to underscore. +assert_klp_sym other_mod_func kvm_intel + +pass "Module.symvers paths normalized to runtime module names" diff --git a/tools/objtool/tests/generic/test-module-object.sh b/tools/objtool/tests/generic/test-module-object.sh new file mode 100755 index 00000000000000..95c8402a523a21 --- /dev/null +++ b/tools/objtool/tests/generic/test-module-object.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A klp relocation section is named for the object being patched, not for the +# object which happens to own the symbol being referenced. Deriving it from +# the symbol means a cross-module reference lands in a section for an object +# the patch may not even touch, so the relocation is never applied and the call +# goes somewhere arbitrary. + +. "$(dirname "$0")/../lib.sh" + +setup +build_module_pair cross_module.c klp_testmod + +# The fixture has to have built as a module for any of this to mean anything. +in_sections orig.o | grep -q '\.modinfo' || + fail "fixture has no .modinfo" + +# other_mod_func belongs to a different module than the one being patched. +add_exports other_mod other_mod_func +run_diff + +# Named for the patched object ... +assert_section __klp_relocs.klp_testmod +# ... not for the object owning the symbol. +assert_no_section __klp_relocs.other_mod + +run_post_link +assert_klp_rela klp_testmod .text.target + +pass "klp relocation section named for the patched object" diff --git a/tools/objtool/tests/generic/test-module-vmlinux-reloc.sh b/tools/objtool/tests/generic/test-module-vmlinux-reloc.sh new file mode 100755 index 00000000000000..635a75f6b8ba63 --- /dev/null +++ b/tools/objtool/tests/generic/test-module-vmlinux-reloc.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Patching a module, where the patched code references a vmlinux symbol which +# needs a klp relocation. +# +# The kernel does not allow a module-targeted klp relocation to reference a +# vmlinux symbol, and a symbol exported with EXPORT_SYMBOL_FOR_MODULES gets a +# klp relocation. Put together, filing that relocation under the patched +# module produces a patch the kernel refuses to apply to its target. +# +# So it goes under vmlinux instead, and is applied when the patch module loads +# rather than when the patched module does. That is the opposite of the rule +# for a reference to a module's symbol, which test-module-object covers; this +# is the other branch of the same decision. + +. "$(dirname "$0")/../lib.sh" + +setup +# The object being patched is a module ... +build_module_pair cross_module.c klp_testmod + +# ... and the symbol it references belongs to vmlinux, exported in a way that +# still requires a klp relocation. +export_syms +add_exports_ns vmlinux module:kvm other_mod_func +run_diff + +# Filed against vmlinux, applied when the patch loads. +assert_section __klp_relocs.vmlinux +assert_klp_sym other_mod_func vmlinux + +# Not against the patched module: that is the relocation the kernel rejects. +assert_no_section __klp_relocs.klp_testmod + +run_post_link +assert_klp_rela vmlinux .text.target +assert_no_section ".klp.rela.klp_testmod..text.target" + +pass "klp relocation to a vmlinux symbol filed under vmlinux, not the patched module" diff --git a/tools/objtool/tests/generic/test-new-data.sh b/tools/objtool/tests/generic/test-new-data.sh new file mode 100755 index 00000000000000..acb68ff19c998f --- /dev/null +++ b/tools/objtool/tests/generic/test-new-data.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Data added by the patch has no counterpart in the running kernel and must be +# carried into the livepatch. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair new_data.c + +# State the premise on both sides. The array is new in the patched build and +# absent from the original; if the compiler folded it into the code instead of +# emitting it, the assertion below would fail without saying why. +has_input_symbol "$orig_obj" klp_new_data && + fail "fixture put klp_new_data in the original; nothing new to carry" +has_input_symbol "$patched_obj" klp_new_data || + fail "compiler did not emit klp_new_data; the fixture tests nothing" +in_relocs "$patched_obj" | grep -q 'klp_new_data' || + fail "target() does not reference klp_new_data; the fixture tests nothing" + +run_diff + +assert_patched target +assert_symbol klp_new_data + +pass "new data carried into the patch" diff --git a/tools/objtool/tests/generic/test-new-export-ref.sh b/tools/objtool/tests/generic/test-new-export-ref.sh new file mode 100755 index 00000000000000..f0be2cf87fe3c4 --- /dev/null +++ b/tools/objtool/tests/generic/test-new-export-ref.sh @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A reference the patch adds has no counterpart in the original object. klp +# diff used to reject any such reference needing a klp relocation, which ruled +# out patches that call something they did not call before -- a common enough +# thing for a fix to do. +# +# Module.symvers is what makes it safe: it says the symbol exists and who owns +# it. But that is only sufficient for a vmlinux export. A new reference to a +# module's export is a dependency the patch module does not declare, and the +# relocation would resolve only if that module happened to be loaded, so it +# stays an error. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair new_export_ref.c + +# Exported by vmlinux, in a module: namespace so it needs a klp relocation +# rather than an ordinary one. Allowed. +export_syms +add_exports_ns vmlinux module:kvm newly_referenced +run_diff +assert_klp_sym newly_referenced vmlinux + +# Exported by a module the patched object does not depend on. Rejected, and +# for that reason rather than some other. +export_syms +add_exports other_mod newly_referenced +run_diff 255 +assert_diff_log 'undeclared module dependency' + +# ... unless the original already referenced something that module exports. +# The loader will not let the patched object load without other_mod, so the +# klp relocation has something to resolve against, and klp diff allows it. +# This is the other half of the rule, and it fails in the opposite direction: +# refusing here would reject a patch which is safe to apply. +rm -f "$workdir/out.o" +build_pair new_export_ref.c -DEXISTING_DEP +export_syms +add_exports other_mod newly_referenced existing_dep +run_diff +assert_klp_sym newly_referenced other_mod + +pass "new reference allowed for vmlinux and for a module already depended on" diff --git a/tools/objtool/tests/generic/test-new-function.sh b/tools/objtool/tests/generic/test-new-function.sh new file mode 100755 index 00000000000000..b0b7f644311075 --- /dev/null +++ b/tools/objtool/tests/generic/test-new-function.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A function added by the patch has no original to correlate against and must +# still be carried into the livepatch. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair new_function.c +run_diff + +assert_patched target +assert_symbol klp_new_helper + +pass "new function carried into the patch with its caller" diff --git a/tools/objtool/tests/generic/test-post-link.sh b/tools/objtool/tests/generic/test-post-link.sh new file mode 100755 index 00000000000000..a6d40ae1b18d1b --- /dev/null +++ b/tools/objtool/tests/generic/test-post-link.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# klp post-link converts the intermediate __klp_relocs.* sections into the +# .klp.rela.* form the kernel applies at patch load. Getting this wrong is +# invisible at build time: the module links and loads, and the relocations are +# simply never applied. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_local.c + +# An unexported symbol is what produces a klp relocation in the first place. +# The static local is renamed by the compiler -- counter.0 with gcc -- so the +# fixture's premise is that some local of that shape exists, not that a symbol +# called "counter" does. Find it first; everything below names it. +local_sym="$(in_symbols orig.o | + awk '$4 == "OBJECT" && $5 == "LOCAL" && $8 ~ /counter/ { print $8; exit }')" +[ -n "$local_sym" ] || fail "fixture produced no local 'counter' symbol" + +run_diff +assert_section __klp_relocs.vmlinux + +# Nothing has converted them yet. +assert_no_section ".klp.rela.vmlinux..text.target" + +# The original relocation is neutralised by pointing it at a tombstone, which +# is what stops the module loader resolving it behind livepatch's back. +# +# Compilers mangle a static local differently -- gcc says counter.0, clang +# target.counter -- so find what this one produced rather than assuming. +assert_tombstone "$local_sym" + +run_post_link + +# One .klp.rela section per base section, carrying SHF_RELA_LIVEPATCH, against +# a symbol in SHN_LIVEPATCH for the kernel to resolve. +assert_klp_rela vmlinux .text.target +assert_livepatch_sym "$local_sym" + +pass "klp relocations converted to .klp.rela with SHN_LIVEPATCH symbols" diff --git a/tools/objtool/tests/generic/test-special-section-shared.sh b/tools/objtool/tests/generic/test-special-section-shared.sh new file mode 100755 index 00000000000000..05c6110b6aef0d --- /dev/null +++ b/tools/objtool/tests/generic/test-special-section-shared.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Only the patched function's special section entry may be extracted. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair special_section_shared.c + +has_input_section orig.o .kcfi_traps || + probe_skip "fixture produced no .kcfi_traps on this arch" + +run_diff + +assert_patched target +assert_not_patched other +assert_section ".kcfi_traps" + +entries="$(out_relocs | awk '/rela\.kcfi_traps/,/^$/' | grep -c 'target')" +[ "$entries" = 1 ] || fail "expected one .kcfi_traps entry, found $entries" + +out_relocs | awk '/rela\.kcfi_traps/,/^$/' | grep -q 'other' && + fail "the untouched function's entry was dragged in" + +pass "only the patched function's entry extracted" diff --git a/tools/objtool/tests/generic/test-special-section.sh b/tools/objtool/tests/generic/test-special-section.sh new file mode 100755 index 00000000000000..b6a9139c0062e1 --- /dev/null +++ b/tools/objtool/tests/generic/test-special-section.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A .kcfi_traps entry belonging to a patched function must be extracted even +# without ANNOTATE_DATA_SPECIAL and with a local label already at offset 0. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair special_section.c + +in_symbols orig.o | grep -q 'trap_marker' || + probe_skip "fixture produced no .kcfi_traps on this arch" + +run_diff + +assert_patched target +assert_section ".kcfi_traps" + +pass ".kcfi_traps extracted despite a local label at offset 0" diff --git a/tools/objtool/tests/generic/test-static-call-annotate-stripped.sh b/tools/objtool/tests/generic/test-static-call-annotate-stripped.sh new file mode 100755 index 00000000000000..818b047ed4b62f --- /dev/null +++ b/tools/objtool/tests/generic/test-static-call-annotate-stripped.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A patch may remove the last ANNOTATE_DATA_SPECIAL in a translation unit while +# leaving the special section it described in place. +# +# klp diff needs entry boundaries for a special section: either an entsize, or +# annotations naming where each entry starts. .static_call_sites has no +# entsize, so the annotations are all there is -- and when the patched object +# is the only side that lost them, the two sides no longer agree on how the +# section divides up. +# +# The section must still be handled. Dropping it would leave the patched +# function's static call unregistered; misreading its boundaries would attach +# the entry to the wrong code. Either way nothing is reported at build time. +# +# Fixed by commit 3de711fba73a ("objtool/klp: Fix create_fake_symbols() +# skipping entsize-based sections"). +# +# Covers the same ground as corpus/x86_64/static-call-annotate-stripped in Joe +# Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_call.c -DNO_ANNOTATE + +# The premise: the original describes its entry, the patched one no longer +# does, and both still have the section itself. +has_input_section orig.o .discard.annotate_data || + fail "fixture produced no annotation in the original" +has_input_section patched.o .discard.annotate_data && + fail "patched object still has the annotation; nothing was stripped" +assert_input_section .static_call_sites + +run_diff + +assert_patched target +assert_section .static_call_sites +assert_reloc_sym .static_call_sites target + +pass "static call site kept when the patch strips its data annotation" diff --git a/tools/objtool/tests/generic/test-static-call-module-key.sh b/tools/objtool/tests/generic/test-static-call-module-key.sh new file mode 100755 index 00000000000000..260c4af72e051c --- /dev/null +++ b/tools/objtool/tests/generic/test-static-call-module-key.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# As for static branches, a static call key owned by a module must be rejected +# while a vmlinux-owned one is accepted. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_call.c + +has_input_section orig.o .static_call_sites || + probe_skip "fixture produced no .static_call_sites on this arch" + +run_diff +assert_patched target + +# The accepted half has to show the entry was carried, not just that the +# function was: dropping the section silently would leave the patched call +# unregistered, and "target was cloned" cannot tell the two apart. +assert_section .static_call_sites +assert_reloc_sym .static_call_sites target + +rm -f "$workdir/out.o" +build_pair static_call.c -DMODNAME='"klp_testmod"' +run_diff 255 + +diff_log | grep -q 'unsupported static call key __SCK__klp_test_call' || + fail "expected rejection, got: $(diff_log | tail -1)" + +# A rejection has to leave nothing behind. out.o was removed above, so +# anything here was written by the run which was supposed to refuse. +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "module-owned static call key rejected, vmlinux-owned accepted" diff --git a/tools/objtool/tests/generic/test-static-call-new.sh b/tools/objtool/tests/generic/test-static-call-new.sh new file mode 100755 index 00000000000000..f7d2b39c0fec2c --- /dev/null +++ b/tools/objtool/tests/generic/test-static-call-new.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A patch may introduce a static call where the original function had none. +# +# The .static_call_sites entry is then new, with nothing in the original to +# correlate it against, so klp diff has to carry it into the patch from +# scratch. Where the key lives still decides whether that is allowed: a +# vmlinux key is reachable, and a module-owned one is not, for the same reason +# an existing module key is not -- late module patching lets the livepatch load +# first, and the unresolved entry is dereferenced when the module arrives. +# +# Both halves are here because they fail in opposite directions. Dropping the +# new entry leaves a static call the kernel never patches; accepting a new +# module-owned one is the corruption the check exists to prevent. +# +# Covers the same ground as corpus/x86_64/static-call-vmlinux-new and +# static-call-module-new in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_call.c -DNEW_CALL + +# The premise: the original really has no static call, the patched one does. +has_input_section orig.o .static_call_sites && + fail "fixture put a .static_call_sites in the original; nothing new to add" +has_input_section patched.o .static_call_sites || + probe_skip "compiler produced no .static_call_sites on this arch" + +run_diff +assert_patched target +assert_section .static_call_sites +assert_reloc_sym .static_call_sites target + +# The same new call, with the key owned by a module: not reachable, so the +# build has to stop rather than emit a relocation nothing will resolve. +rm -f "$workdir/out.o" +build_pair static_call.c -DNEW_CALL -DMODNAME='"klp_testmod"' +run_diff 255 +assert_diff_log 'unsupported static call key __SCK__klp_test_call' +[ -e "$workdir/out.o" ] && + fail "output object produced for a rejected input" + +pass "static call introduced by the patch carried in, or rejected for a module key" diff --git a/tools/objtool/tests/generic/test-static-local-uncorrelated.sh b/tools/objtool/tests/generic/test-static-local-uncorrelated.sh new file mode 100755 index 00000000000000..d7711587d2d4a6 --- /dev/null +++ b/tools/objtool/tests/generic/test-static-local-uncorrelated.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Some static locals must not be correlated with their counterparts in the +# running kernel; the patched code has to use a fresh copy instead. +# +# .data..once holds the "have we warned yet" flags behind WARN_ONCE. Correlate +# one and the patched function inherits the flag from before the patch, so the +# warning the patch was written to produce never fires. The same goes for the +# names the kernel generates for per-instance state -- __warned, __key, +# __func__ and friends. +# +# Both directions matter, so an ordinary static local is here too: a rule that +# refuses to correlate anything would pass a test that only checks the +# refusals. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_local_uncorrelated.c +run_diff + +# Compilers mangle static locals differently -- gcc gives __key.1, Clang +# target.__key -- so match on the base name. + +# Correlated: referenced through a klp symbol, pointing at the kernel's copy. +out_symbols | grep -q '\.klp\.sym\..*ordinary' || + fail "ordinary static local was not correlated" + +# Not correlated: no klp symbol, and a copy cloned into the patch instead. +out_symbols | grep -q '\.klp\.sym\..*__key' && + fail "__key was correlated; it must use a fresh copy" +# .sbss/.sdata on the architectures with a small-data area. +out_sections | grep -qE '\.s?(bss|data)[^ ]*__key' || + fail "__key was neither correlated nor cloned" + +out_symbols | grep -q '\.klp\.sym\..*once_flag' && + fail ".data..once variable was correlated; it must use a fresh copy" +assert_section '.data..once' + +pass "per-instance static locals cloned, ordinary ones correlated" diff --git a/tools/objtool/tests/generic/test-static-local.sh b/tools/objtool/tests/generic/test-static-local.sh new file mode 100755 index 00000000000000..d57efa64dfd92a --- /dev/null +++ b/tools/objtool/tests/generic/test-static-local.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A static local must be correlated with the original, not duplicated: a second +# copy would discard the state the running kernel accumulated. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair static_local.c + +in_symbols orig.o | grep -q 'counter' || + probe_skip "compiler emitted no distinct static local symbol" + +run_diff +assert_patched target + +out_symbols | grep -q '\.klp\.sym\..*\.counter' || + fail "static local not referenced through a klp relocation" + +out_symbols | grep 'counter' | grep -qvE 'UND|\.klp\.(sym|tombstone)' && + fail "static local was given a fresh definition" + +pass "static local correlated rather than duplicated" diff --git a/tools/objtool/tests/generic/test-switch-rodata.sh b/tools/objtool/tests/generic/test-switch-rodata.sh new file mode 100755 index 00000000000000..fb27e96c65f680 --- /dev/null +++ b/tools/objtool/tests/generic/test-switch-rodata.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A Clang switch jump table travels with the function it belongs to. +# +# For a dense enough switch Clang emits the targets as a table in +# .rodata..Lswitch.table., named after the function but not part of +# it. klp diff has to associate the two: the patched function indexes into +# that table, so a clone which does not bring it along jumps through whatever +# the kernel's copy holds -- which, when the patch changed the switch, is the +# wrong set of targets. +# +# That is an indirect jump to a stale address, not a missing symbol, so nothing +# reports it at build or load time. +# +# objtool has no switch-specific code: the table is carried by the general +# mechanism for data a cloned function references. So this is a regression +# test on that mechanism reaching a shape it is easy to get wrong, not a guard +# on a particular line -- making the table uncorrelated, the nearest sabotage, +# does not change the outcome. +# +# Covers the same ground as corpus/x86_64-llvm-switch-rodata/ +# clang-switch-rodata-assoc in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +clang_only "only Clang emits switch jump tables in their own section" + +setup +build_pair switch_rodata.c + +# The premise: this Clang really did build a table rather than a chain of +# comparisons, and the added case really did change it. +tbl=.rodata..Lswitch.table.status_to_string +has_input_section orig.o "$tbl" || + probe_skip "this clang built no jump table for the switch" +# readelf prefixes each line with "[nn]", which splits into one or two fields +# depending on the index, so strip it before counting columns. +tbl_size() +{ + in_sections "$1" | sed 's/^ *\[[ 0-9]*\] *//' | + awk -v s="$tbl" '$1 == s { print $5 }' +} +[ "$(tbl_size orig.o)" != "$(tbl_size patched.o)" ] || + fail "fixture's added case did not change the jump table" + +run_diff + +assert_patched status_to_string +assert_section "$tbl" +assert_reloc_sym .text.status_to_string "$tbl" + +pass "Clang switch jump table carried with the function it belongs to" diff --git a/tools/objtool/tests/generic/test-symid-discarded.sh b/tools/objtool/tests/generic/test-symid-discarded.sh new file mode 100755 index 00000000000000..388a24984ec0a3 --- /dev/null +++ b/tools/objtool/tests/generic/test-symid-discarded.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# .klp.symid must not reference symbols in sections the vmlinux link discards. +# Each such section has been its own bug, found only when someone built a +# config where a duplicate happened to land there, so cover the whole list +# rather than whichever one was reported last. + +. "$(dirname "$0")/../lib.sh" + +setup + +# Allocated sections which vmlinux.lds.h discards unconditionally. A symid +# referencing one of these fails the vmlinux link outright: +# +# `__exitcall_hid_exit' referenced in section `.klp.symid' of vmlinux.o: +# defined in discarded section `.exitcall.exit' of vmlinux.o +for sec in .exitcall.exit .no_trim_symbol; do + build_one symid_discarded.c a.o \ + -DFUNC_NAME=use_a -DDISCARDED_SEC="\"$sec\"" + build_one symid_discarded.c b.o \ + -DFUNC_NAME=use_b -DDISCARDED_SEC="\"$sec\"" + + # --klp-symids only runs on a file named vmlinux.o + rm -f "$workdir/vmlinux.o" + partial_link "$workdir/vmlinux.o" "$workdir/a.o" "$workdir/b.o" || + probe_skip "partial link unavailable" + + "$OBJTOOL" --klp-symids --link "$workdir/vmlinux.o" || + fail "objtool --klp-symids failed" + + symids="$(in_relocs vmlinux.o | + awk '/rela.klp.symid/,/^$/')" + + # Without this the test would also pass if symid generation stopped + # entirely. + echo "$symids" | grep -q 'dup_normal' || + fail "$sec: no symid for the duplicate in a live section" + + echo "$symids" | grep -q 'dup_discarded' && + fail "symid emitted for a symbol in discarded section $sec" +done + +pass "no symids for symbols in discarded sections" diff --git a/tools/objtool/tests/generic/test-sympos-vmlinux.sh b/tools/objtool/tests/generic/test-sympos-vmlinux.sh new file mode 100755 index 00000000000000..b2b44001446f8a --- /dev/null +++ b/tools/objtool/tests/generic/test-sympos-vmlinux.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# sympos for vmlinux, which is resolved differently from sympos for a module. +# +# A module's .ko preserves symbol table order, so klp diff can count -- that is +# what test-sympos covers. vmlinux cannot be counted: the final link reorders +# sub-sections, so the order in vmlinux.o is not the order the running kernel +# has. klp diff bridges that with .klp.symid, a table of { id, address } +# emitted into vmlinux.o whose addresses the linker resolves, read back out of +# the linked vmlinux. +# +# Getting it wrong points the relocation at a different symbol of the same +# name. Nothing fails to build or load; the patched code uses the wrong +# object. +# +# The fixture is arranged so the two answers differ: the static that comes +# first in the symbol table is placed at the *higher* address, so counting +# gives 1 and reading the linked image gives 2. Without that, both paths agree +# and the test cannot tell them apart. + +. "$(dirname "$0")/../lib.sh" + +setup + +# use_a's static sorts last by section name, use_b's first. Only use_a is +# patched, so exactly one sympos comes out. +build_one sympos_vmlinux.c orig_a.o -DFUNC_NAME=use_a -DVARSEC='".data.zzz"' +build_one sympos_vmlinux.c patched_a.o -DFUNC_NAME=use_a -DVARSEC='".data.zzz"' -DPATCHED +build_one sympos_vmlinux.c b.o -DFUNC_NAME=use_b -DVARSEC='".data.aaa"' -DNO_MODINFO + +make_vmlinux_pair "$workdir/orig_a.o" "$workdir/b.o" \ + -- "$workdir/patched_a.o" "$workdir/b.o" + +[ "$(count_input_symbols vmlinux.o dup_counter)" = 2 ] || + fail "fixture did not produce two dup_counter symbols" +has_input_section vmlinux.o .klp.symid || + fail "objtool --klp-symids emitted no .klp.symid table" +has_input_section vmlinux .klp.symid || + fail ".klp.symid did not survive the link" + +# The premise: symbol table order and address order must disagree, or the test +# proves nothing. +first_addr="$(in_symbols vmlinux | awk '$8 == "dup_counter" { print $2; exit }')" +low_addr="$(in_symbols vmlinux | awk '$8 == "dup_counter" { print $2 }' | sort | head -1)" +[ "$first_addr" != "$low_addr" ] || + probe_skip "linker did not reorder the two statics" + +assert_input_symbol dup_counter +run_diff + +# Address order says 2. Counting symbol table order would say 1. +assert_klp_sympos dup_counter 2 +out_symbols | grep -q 'dup_counter,1' && + fail "sympos 1 emitted: counted symbol table order instead of reading the linked vmlinux" + +pass "vmlinux sympos taken from the linked image, not from symbol table order" diff --git a/tools/objtool/tests/generic/test-sympos.sh b/tools/objtool/tests/generic/test-sympos.sh new file mode 100755 index 00000000000000..b71d4930a22af6 --- /dev/null +++ b/tools/objtool/tests/generic/test-sympos.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# sympos is what livepatch uses to tell duplicate symbol names apart in the +# patched object: which "dup_counter" of several the relocation means. Get it +# wrong and the patch resolves to the wrong object at load time, silently. +# +# klp_find_sympos() reports 0 when a name is unique and a 1-based position when +# it is not, so both need checking -- always reporting a position, or never, +# each looks right in one of the two cases. +# +# This is the module path, counting symbol table order. vmlinux is reordered +# by the final link and goes through .klp.symid instead; that needs a linked +# vmlinux next to vmlinux.o and is not covered here. + +. "$(dirname "$0")/../lib.sh" + +setup + +# One copy: the name is unique, so there is nothing to disambiguate. +build_one sympos_dup.c orig.o -DFUNC_NAME=use_a +build_one sympos_dup.c patched.o -DFUNC_NAME=use_a -DPATCHED +run_diff + +assert_klp_sympos dup_counter 0 + +# Two copies: positions, in symbol table order. +for p in "" "-DPATCHED"; do + # shellcheck disable=SC2086 + build_one sympos_dup.c "a$p.o" -DFUNC_NAME=use_a $p + # shellcheck disable=SC2086 + build_one sympos_dup.c "b$p.o" -DFUNC_NAME=use_b -DNO_MODINFO $p +done +partial_link "$workdir/orig.o" "$workdir/a.o" "$workdir/b.o" || + probe_skip "partial link unavailable" +partial_link "$workdir/patched.o" "$workdir/a-DPATCHED.o" "$workdir/b-DPATCHED.o" || + probe_skip "partial link unavailable" + +# Without duplicates in the input there is nothing for sympos to number. +[ "$(count_input_symbols orig.o dup_counter)" = 2 ] || + fail "fixture did not produce two dup_counter symbols" + +run_diff + +assert_klp_sympos dup_counter 1 +assert_klp_sympos dup_counter 2 +# ... and nothing still claiming the name is unique +out_symbols | grep -qE '\.klp\.sym\.[^.]+\.dup_counter,0([[:space:]]|$)' && + fail "sympos 0 emitted for a duplicated symbol" + +pass "sympos numbers duplicate symbols and stays 0 for unique ones" diff --git a/tools/objtool/tests/generic/test-symvers-parse-error.sh b/tools/objtool/tests/generic/test-symvers-parse-error.sh new file mode 100755 index 00000000000000..d597f28e561752 --- /dev/null +++ b/tools/objtool/tests/generic/test-symvers-parse-error.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A malformed Module.symvers has to be reported against the line it is on. +# Module.symvers has tens of thousands of lines and is generated, so a wrong +# line number sends whoever has to fix it to the wrong place, and "line 1" is +# wrong in a way that looks plausible. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair basic.c + +# Three well-formed lines, then one with no tabs at all. +export_syms a b c +echo 'this line has no fields' >> "$workdir/Module.symvers" + +run_diff 255 + +assert_diff_log 'malformed Module.symvers' +assert_diff_log 'at line 4' + +pass "malformed Module.symvers reported against the offending line" diff --git a/tools/objtool/tests/generic/test-thinlto-ambiguity.sh b/tools/objtool/tests/generic/test-thinlto-ambiguity.sh new file mode 100755 index 00000000000000..34f4f3a58e2013 --- /dev/null +++ b/tools/objtool/tests/generic/test-thinlto-ambiguity.sh @@ -0,0 +1,77 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Two ThinLTO-promoted symbols sharing a demangled name must be paired up +# correctly. +# +# A file-local symbol which ThinLTO has to make visible is renamed +# helper.llvm.. With two such helpers in one link the original and the +# patched object hold two each, all four spelled differently, and demangling +# gives "helper" for all of them -- so the name is not enough to say which +# corresponds to which. +# +# Getting it wrong is silent and specific: the patch is built against the wrong +# body, so one call site gets the other helper's arithmetic. Nothing fails to +# build and nothing fails to load. +# +# test-thinlto-local covers the unambiguous case, one promoted symbol whose +# hash moved. This is the case where demangling alone is not an answer. +# +# The outcome is asserted, not the machinery: with the clang tested here the +# pairing succeeds even with the .llvm. suffix map disabled and with +# llvm_suffix() stubbed out, so no single-line sabotage distinguishes it. The +# tiered matcher this case was written for is not needed for this shape. +# +# Covers the same ground as corpus/x86_64-llvm-thinlto/ +# thin-lto-demangled-ambiguity and thin-lto-demangled-global-match in Joe +# Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +clang_only "ThinLTO requires clang" + +find_thinlto_toolchain || + probe_skip "no matching clang/lld pair for a ThinLTO link; set THIN_LD to one" + +build_thinlto() # $1 output object, $2 extra flags +{ + local t + for t in "" -DTU_B -DTU_C; do + $THIN_CC -flto=thin -O2 -ffunction-sections -fdata-sections \ + $2 $t -c "$FIXTURES_DIR/thinlto_ambiguity.c" \ + -o "$workdir/tu$t.o" 2>/dev/null || return 1 + done + "$THIN_LD" -r "$workdir/tu.o" "$workdir/tu-DTU_B.o" "$workdir/tu-DTU_C.o" \ + -o "$1" 2>/dev/null || return 1 +} + +build_thinlto "$workdir/orig.o" "" || + probe_skip "ThinLTO build failed ($THIN_CC, $THIN_LD)" +build_thinlto "$workdir/patched.o" -DPATCHED || + probe_skip "ThinLTO build failed ($THIN_CC, $THIN_LD)" + +# The premise: two promoted helpers per object, and exactly one of them kept +# its hash -- the one the patch did not touch. Without that there is nothing +# to disambiguate. +orig_syms="$(in_symbols orig.o | grep -oE 'helper\.llvm\.[0-9]+' | sort -u)" +new_syms="$( in_symbols patched.o | grep -oE 'helper\.llvm\.[0-9]+' | sort -u)" +[ "$(echo "$orig_syms" | wc -l)" = 2 ] && [ "$(echo "$new_syms" | wc -l)" = 2 ] || + probe_skip "ThinLTO did not promote two distinct helpers here" + +kept="$(comm -12 <(echo "$orig_syms") <(echo "$new_syms"))" +moved="$(comm -13 <(echo "$orig_syms") <(echo "$new_syms"))" +[ "$(echo "$kept" | wc -w)" = 1 ] && [ "$(echo "$moved" | wc -w)" = 1 ] || + probe_skip "expected one helper to keep its hash and one to move" + +run_diff + +# Exactly one helper is cloned, and it is the one whose body changed. Cloning +# the other, or both, is what a wrong pairing looks like. +assert_not_patched "$kept" + +n="$(out_sections | grep -cE '[[:space:]]\.text\.helper\.llvm\.[0-9]+[[:space:]]')" +[ "$n" = 1 ] || + fail "expected 1 cloned helper, found $n" + +pass "ThinLTO helpers sharing a demangled name paired up correctly" diff --git a/tools/objtool/tests/generic/test-thinlto-local.sh b/tools/objtool/tests/generic/test-thinlto-local.sh new file mode 100755 index 00000000000000..a266263676ed5f --- /dev/null +++ b/tools/objtool/tests/generic/test-thinlto-local.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Correlating ThinLTO-promoted locals requires demangling the .llvm. +# suffix, and the resulting klp relocation must name the original symbol: that +# is the one in the running kernel's kallsyms. + +. "$(dirname "$0")/../lib.sh" + +setup +clang_only "ThinLTO requires clang" + +find_thinlto_toolchain || + probe_skip "no matching clang/lld pair for a ThinLTO link; set THIN_LD to one" + +build_thinlto() # $1 output object, $2 extra flags +{ + $THIN_CC -flto=thin -O2 -ffunction-sections -fdata-sections $2 \ + -c "$FIXTURES_DIR/thinlto_local.c" -o "$workdir/tu_a.o" 2>/dev/null || return 1 + $THIN_CC -flto=thin -O2 -ffunction-sections -fdata-sections $2 -DTU_B \ + -c "$FIXTURES_DIR/thinlto_local.c" -o "$workdir/tu_b.o" 2>/dev/null || return 1 + "$THIN_LD" -r "$workdir/tu_a.o" "$workdir/tu_b.o" -o "$1" 2>/dev/null || return 1 +} + +build_thinlto "$workdir/orig.o" "" || + probe_skip "ThinLTO build failed ($THIN_CC, $THIN_LD)" +build_thinlto "$workdir/patched.o" -DPATCHED || + probe_skip "ThinLTO build failed ($THIN_CC, $THIN_LD)" + +orig_sym="$(in_symbols orig.o | grep -o 'counter\.llvm\.[0-9]*' | head -1)" +new_sym="$( in_symbols patched.o | grep -o 'counter\.llvm\.[0-9]*' | head -1)" + +[ -n "$orig_sym" ] && [ -n "$new_sym" ] || + probe_skip "$THIN_CC did not promote the local symbol" + +# Equal hashes would make plain name matching work, testing nothing. +[ "$orig_sym" != "$new_sym" ] || + probe_skip "$THIN_CC gave the same ThinLTO hash for both builds" + +run_diff +assert_patched target + +out_symbols | grep -q "\.klp\.sym\.vmlinux\.$orig_sym," || + fail "expected a klp relocation naming $orig_sym" +out_symbols | grep -q "\.klp\.sym\.vmlinux\.$new_sym," && + fail "klp relocation names $new_sym, which the running kernel does not have" + +pass "ThinLTO-mangled local correlated across differing hashes ($THIN_CC, $THIN_LD)" diff --git a/tools/objtool/tests/generic/test-ubsan-noise.sh b/tools/objtool/tests/generic/test-ubsan-noise.sh new file mode 100755 index 00000000000000..b415eb16bfd22c --- /dev/null +++ b/tools/objtool/tests/generic/test-ubsan-noise.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# UBSAN instrumentation in an unchanged function must not make it look changed. +# +# Every instrumented operation gets a per-callsite metadata object in an +# anonymous data section -- .data..Lubsan_data and .data..Lubsan_type from GCC, +# .data..L__unnamed_ from Clang -- whose names are compiler-generated and mean +# nothing across a rebuild. is_uncorrelated_section() exists so klp diff does +# not try to pair them up. +# +# Without that, the metadata belonging to a function nobody touched compares as +# different and drags the function into the patch. A livepatch which replaces +# functions the patch never changed is not a build failure: it is a larger +# patch than intended, taking its dependencies with it, and every extra +# function is one more that can fail to correlate or to apply. +# +# Covers the same ground as corpus/x86_64-ubsan/{ubsan-shift-noise, +# ubsan-metadata-data-section,gcc-ubsan-anonymous-data,ubsan-handler-cloning} +# and corpus/x86_64-llvm-ubsan/{clang-ubsan-bounds-noise, +# clang-ubsan-handler-cloning} in Joe Lawrence's klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair ubsan_noise.c -fsanitize=shift + +# The premise: this compiler really did instrument, and left its metadata in an +# anonymous section. Without that the test is just test-basic again. +ubsan_sec="$(in_sections orig.o | + grep -oE '\.data\.\.L(ubsan_data|__unnamed_)[A-Za-z0-9_.]*' | head -1)" +[ -n "$ubsan_sec" ] || + probe_skip "compiler emitted no anonymous UBSAN data section" +assert_input_symbol untouched + +run_diff + +# The changed function is patched, and the untouched one is left alone despite +# carrying instrumentation of its own. +assert_patched touched +assert_not_patched untouched + +# The handler the patched code calls has to come with it, or the clone calls +# nothing when its check fires. +out_symbols | grep -q '__ubsan_handle_' || + fail "no __ubsan_handle_* reference in the patched output" + +pass "UBSAN metadata in an unchanged function does not drag it into the patch" diff --git a/tools/objtool/tests/lib.sh b/tools/objtool/tests/lib.sh new file mode 100644 index 00000000000000..4da168d0ddca31 --- /dev/null +++ b/tools/objtool/tests/lib.sh @@ -0,0 +1,911 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Helpers for the objtool klp tests. A test builds a fixture twice, as the +# original and (with -DPATCHED) the patched object, runs both through +# "klp checksum" and diffs them, then asserts on the result. +# +# Assertions check properties rather than compare against recorded output: +# codegen varies between compilers and golden files would report churn instead +# of regressions. + +set -u + +TESTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Tests live in generic/ or in an architecture directory beside it, and each +# carries its own fixtures. +FIXTURES_DIR="$(cd "$(dirname "$0")/fixtures" 2>/dev/null && pwd)" + +# The kernel's convention: CROSS_COMPILE is the one knob, with per-tool +# overrides for what it does not cover. objtool itself is always a host binary +# -- it is built with HOSTCC and only reads ELF -- so an arm64 machine can run +# the x86 tests against x86 objects given a compiler that emits them. +# +# readelf reads any target, so it rarely needs overriding, and either GNU +# readelf or llvm-readelf will do: the assertions match on fields rather than +# on columns, and where the two spell something differently -- "OS [0xff20]" +# against "OS[0xff20]" for SHN_LIVEPATCH -- they accept both. BFD's objcopy is +# usually built for the host's target alone, and llvm-objcopy is the +# target-agnostic replacement. +CROSS_COMPILE="${CROSS_COMPILE:-}" +CC="${CC:-${CROSS_COMPILE}gcc}" +LD="${LD:-${CROSS_COMPILE}ld}" +READELF="${READELF:-${CROSS_COMPILE}readelf}" +OBJCOPY="${OBJCOPY:-${CROSS_COMPILE}objcopy}" + +OBJTOOL="${OBJTOOL:-$TESTS_DIR/../objtool}" + +# klp_preflight +# +# Check the environment once, before any test runs, and report what was found. +# +klp_preflight() +{ + local tmp tool cc_version arch host cc_arch + + bail() { echo "Bail out! $*" >&2; exit 1; } + + # A relative $OBJTOOL is relative to the objtool directory, not tests/. + [ -x "$OBJTOOL" ] || [ ! -x "$TESTS_DIR/../$OBJTOOL" ] || + OBJTOOL="$TESTS_DIR/../$OBJTOOL" + + [ -x "$OBJTOOL" ] || + bail "objtool not found at '$OBJTOOL' -- build it first" + + # run_diff() runs objtool from inside the test's working directory, so + # a relative path would resolve against that instead. + OBJTOOL="$(realpath "$OBJTOOL")" + + "$OBJTOOL" klp 2>&1 | grep -q checksum || + bail "objtool was built without klp support; install libxxhash (>= 0.8) and rebuild" + + command -v "${CC%% *}" >/dev/null || bail "compiler not found: $CC" + + for tool in "$READELF" "$OBJCOPY" "$LD"; do + command -v "${tool%% *}" >/dev/null || bail "$tool not found" + done + + tmp="$(mktemp -d)" || bail "mktemp failed" + echo 'int probe(void) { return 0; }' > "$tmp/probe.c" + $CC -c -o "$tmp/probe.o" "$tmp/probe.c" 2>/dev/null || + { rm -rf "$tmp"; bail "$CC cannot compile a trivial object"; } + + # $CC, $ARCH and objtool have to agree about the target, and cross runs + # are where they stop agreeing: plain "CC=clang ARCH=x86_64" on an arm64 + # box selects the x86 tests and then builds arm64 objects, because clang + # needs --target= to emit anything but the host's. + # + # Ask objtool rather than comparing machine names. It rejects an object + # it was not built for -- "unexpected ELF machine type" -- so one check + # covers every way the three can disagree, and says so once instead of + # failing every test for the same reason. + "$OBJTOOL" klp checksum "$tmp/probe.o" >/dev/null 2>&1 || + { rm -rf "$tmp" + bail "objtool rejects an object built by '$CC'; they target" \ + "different architectures (set CROSS_COMPILE, or" \ + "--target= for clang)"; } + + # BFD objcopy is usually built for the host's target alone, and + # checksum_of() needs it to read the object under test. + $OBJCOPY -O binary --only-section=.text "$tmp/probe.o" "$tmp/probe.bin" 2>/dev/null || + { rm -rf "$tmp" + bail "$OBJCOPY cannot read objects built by '$CC'; install" \ + "binutils-multiarch or set OBJCOPY=llvm-objcopy"; } + # $ARCH only chooses which directory of tests runs, so it can disagree + # with what $CC builds without objtool noticing -- and the result is the + # wrong set of tests, quietly. + case "$($READELF -hW "$tmp/probe.o" | sed -n 's/.*Machine: *//p')" in + *X86-64*|*Intel*80386*) cc_arch=x86 ;; + *AArch64*) cc_arch=arm64 ;; + *) cc_arch= ;; + esac + rm -rf "$tmp" + + # Normalize to the kernel's SRCARCH. + case "${ARCH:-$(uname -m)}" in + x86_64|i?86) arch=x86 ;; + aarch64*) arch=arm64 ;; + *) arch="${ARCH:-$(uname -m)}" ;; + esac + + case "$(uname -m)" in + x86_64|i?86) host=x86 ;; + aarch64*) host=arm64 ;; + *) host="$(uname -m)" ;; + esac + + [ -z "$cc_arch" ] || [ "$cc_arch" = "$arch" ] || + bail "ARCH says $arch but '$CC' builds $cc_arch objects;" \ + "the $arch tests would run against the wrong architecture" + + KLP_TEST_ARCH="$arch" + KLP_TEST_PREFLIGHT=done + export OBJTOOL CC KLP_TEST_ARCH KLP_TEST_PREFLIGHT + + cc_version="$($CC --version 2>/dev/null | head -1)" + cat < "$workdir/Module.symvers" + add_exports vmlinux "$@" +} + +# add_exports [symbol...] +# +# Append exports owned by one object, without clearing what is already there, +# so a test can describe a kernel where several objects export things. +# +# Which object owns a symbol is not cosmetic: a reference to a vmlinux symbol +# is applied when the patch module loads, and a reference to a module's symbol +# when that patched module loads, so klp diff files them in different sections. +add_exports() +{ + local owner="$1"; shift + + add_exports_ns "$owner" "" "$@" +} + +# add_exports_ns [symbol...] +# +# Exports in a symbol namespace, the last field of a Module.symvers line. +# +# A "module:" namespace is EXPORT_SYMBOL_FOR_MODULES(), where the module +# loader grants access by matching the importing module's name against the +# list. A livepatch module is never on that list, so such a symbol has to be +# referenced the way an unexported one is. Ordinary namespaces are not +# special here. +add_exports_ns() +{ + local owner="$1" ns="$2"; shift 2 + + local sym + + for sym in "$@"; do + printf '0x00000000\t%s\t%s\tEXPORT_SYMBOL\t%s\n' \ + "$sym" "$owner" "$ns" >> "$workdir/Module.symvers" + done +} + +# gcc_only / clang_only +gcc_only() +{ + case "$($CC --version 2>/dev/null | head -1)" in + *[Gg][Cc][Cc]*) return 0 ;; + esac + declared_skip "gcc only${1:+: $1}" +} + +clang_only() +{ + case "$($CC --version 2>/dev/null | head -1)" in + *clang*) return 0 ;; + esac + declared_skip "clang only${1:+: $1}" +} + +# build_one [cflags...] +build_one() +{ + local fixture out + fixture="$FIXTURES_DIR/$1" + out="$workdir/$2" + shift 2 + + [ -f "$fixture" ] || fail "missing fixture $fixture" + + # run_checksum only runs once per workdir. A fresh object has no + # checksums in it, so anything built now needs that to happen again. + rm -f "$workdir/.checksummed" + + $CC -c $FIXTURE_CFLAGS "$@" -o "$out" "$fixture" 2>"$workdir/cc.log" || + fail "$(basename "$fixture") does not build: $(tail -1 "$workdir/cc.log")" +} + +# build_pair [cflags...] +build_pair() +{ + local fixture="$1"; shift + + # Name what this builds. A test may run several segments, and + # make_vmlinux_pair() repoints orig_obj at the image it links, so + # without this the next run_diff() would still be reading that. + orig_obj=orig.o + patched_obj=patched.o + + build_one "$fixture" orig.o "$@" + build_one "$fixture" patched.o "$@" -DPATCHED +} + +# run_objtool_check +# +# Run objtool's ordinary check pass over the pair, as the kernel build does. +# +# Some of what klp diff consumes is produced by this pass rather than by the +# compiler: .static_call_sites, .mcount_loc, .ibt_endbr_seal, ORC. +# +# Only module objects see it before klp-build -- with CONFIG_KLP_BUILD the +# per-object pass is deferred, so built-in objects reach klp diff exactly as +# the compiler left them. +run_objtool_check() +{ + local obj + + # This rewrites both objects, so checksums taken before it describe + # something that no longer exists. As in build_one(), drop the marker + # so run_checksum() takes them again. + rm -f "$workdir/.checksummed" + + for obj in "$orig_obj" "$patched_obj"; do + "$OBJTOOL" "$@" "$workdir/$obj" || + fail "objtool $* failed on $obj" + done +} + +run_checksum() +{ + # Checksums live in the objects, and a test may ask for them more than + # once -- diffing the same pair again with a different Module.symvers, + # say. objtool does the right thing when asked twice, leaving the + # object alone, but it says so, and that warning would be most of what + # a passing run prints. Remember instead, and keep quiet. + [ -e "$workdir/.checksummed" ] && return 0 + + "$OBJTOOL" klp checksum "$workdir/$orig_obj" || + fail "klp checksum $orig_obj failed" + "$OBJTOOL" klp checksum "$workdir/$patched_obj" || + fail "klp checksum $patched_obj failed" + touch "$workdir/.checksummed" +} + +# run_diff [expected exit status] +run_diff() +{ + local expect="${1:-0}" rc=0 + + run_checksum + + # klp diff looks for Module.symvers relative to the working directory. + ( cd "$workdir" && "$OBJTOOL" klp diff "$orig_obj" "$patched_obj" out.o ) \ + > "$workdir/diff.log" 2>&1 || rc=$? + + [ "$rc" = "$expect" ] || + fail "klp diff exited $rc, expected $expect: $(tail -2 "$workdir/diff.log")" +} + +cc_supports() +{ + echo 'int f(void) { return 0; }' > "$workdir/flagtest.c" + $CC $1 -c "$workdir/flagtest.c" -o "$workdir/flagtest.o" 2>/dev/null +} + +# partial_link +# +# "ld -r" through the compiler driver so the link targets the same +# architecture as the objects. +partial_link() +{ + local out="$1"; shift + + rm -f "$workdir/.checksummed" + + $CC -r -nostdlib -o "$out" "$@" 2>/dev/null || + $CC -r -nostdlib -fuse-ld=lld -o "$out" "$@" 2>/dev/null +} + +# link_vmlinux +# +# Link objects into an executable, the way the kernel's final link produces +# vmlinux from vmlinux.o. Entry point 0 and no libc: nothing runs it, it only +# has to be a linked image with resolved addresses. +# +# The sub-sections have to come out in name order rather than object order, +# the way the kernel's linker script gathers .text.unlikely and .data.. apart +# from the rest. That reordering is the entire reason .klp.symid exists: a +# link which preserves order cannot tell a correct sympos from one that merely +# counted, and the caller checks the two orders really did diverge. +# +# A linker script rather than --sort-section=name, because lld accepts that +# option and ignores it -- so on a host where only lld can link the target, the +# test would quietly stop testing the thing it is named for. +# +# Three attempts because a cross run has neither $LD nor the compiler's default +# linker able to touch the target: on an arm64 host linking x86 objects, only +# lld will do it. +link_vmlinux() +{ + local out="$1" lds="$workdir/sort.lds"; shift + + echo 'SECTIONS { .data : { *(SORT_BY_NAME(.data.*)) } }' > "$lds" + + $LD -e 0 -T "$lds" -o "$out" "$@" 2>/dev/null || + $CC -nostdlib -Wl,-e,0 -Wl,-T,"$lds" \ + -o "$out" "$@" 2>/dev/null || + $CC -nostdlib -fuse-ld=lld -Wl,-e,0 -Wl,-T,"$lds" \ + -o "$out" "$@" 2>/dev/null +} + +# make_vmlinux_pair -- +# +# Build the vmlinux.o / vmlinux pair klp diff needs to resolve sympos the way +# it does for built-in code, and point the diff at it. +# +# For a module, sympos is a count in symbol table order, which klp diff can do +# from the object alone. vmlinux is different: the final link reorders +# sub-sections, so the position comes from the linked image, bridged by +# .klp.symid. klp diff only looks for that when the object it was handed is +# called vmlinux.o and a vmlinux sits beside it -- so both the name and the +# linked image matter. +make_vmlinux_pair() +{ + local orig=() patched=() seen= arg + + for arg in "$@"; do + if [ "$arg" = -- ]; then seen=y; continue; fi + if [ -n "$seen" ]; then patched+=( "$arg" ); else orig+=( "$arg" ); fi + done + + # Both sides have to have been named. Without this, forgetting the -- + # leaves one list empty, the link of nothing fails, and the test skips + # saying the toolchain cannot link -- which is a test bug wearing the + # costume of an environment one. + [ "${#orig[@]}" -gt 0 ] && [ "${#patched[@]}" -gt 0 ] || + fail "make_vmlinux_pair needs objects either side of --" + + partial_link "$workdir/vmlinux.o" "${orig[@]}" || + probe_skip "partial link unavailable" + partial_link "$workdir/patched.o" "${patched[@]}" || + probe_skip "partial link unavailable" + + "$OBJTOOL" --klp-symids --link "$workdir/vmlinux.o" || + fail "objtool --klp-symids failed" + + link_vmlinux "$workdir/vmlinux" "$workdir/vmlinux.o" || + probe_skip "cannot link a vmlinux here" + + orig_obj=vmlinux.o +} + +# build_module_pair [cflags...] +# +# Build the pair as objects belonging to a module rather than to vmlinux. klp +# diff reads the object's module name from .modinfo, and that decides which +# object a relocation is attributed to and whether a reference counts as +# cross-module, so a good deal of the code has a module path the vmlinux +# fixtures never reach. +# +# The fixture defines its .modinfo name from MODNAME. Passing that through +# -D needs two levels of quoting, which is easy to get wrong at the call site. +build_module_pair() +{ + local fixture="$1" modname="$2"; shift 2 + + build_pair "$fixture" -DMODNAME="\"$modname\"" "$@" +} + +# find_thinlto_toolchain +# +# Set $THIN_LD to an lld from the same LLVM release as $CC (or THIN_CC). A +# mismatched pair fails with "Invalid summary version", which reads like a +# broken test rather than a broken environment. +# +# ThinLTO is clang-only; callers must use clang_only before calling this. +# Only $CC (or an explicit THIN_CC override) is consulted -- the harness does +# not search for a second compiler beside a gcc $CC. +find_thinlto_toolchain() +{ + local cc ver ld + + for cc in "${THIN_CC:-}" "$CC"; do + [ -n "$cc" ] || continue + command -v "${cc%% *}" >/dev/null 2>&1 || return 1 + + ver=$($cc -dumpversion 2>/dev/null | cut -d. -f1) + + for ld in "${THIN_LD:-}" "ld.lld-$ver" ld.lld; do + [ -n "$ld" ] || continue + command -v "$ld" >/dev/null 2>&1 || continue + + echo 'int probe(void) { return 0; }' > "$workdir/probe.c" + $cc -flto=thin -O2 -c "$workdir/probe.c" \ + -o "$workdir/probe.o" 2>/dev/null || continue + "$ld" -r "$workdir/probe.o" -o "$workdir/probe.elf" \ + 2>/dev/null || continue + + THIN_CC="$cc" + THIN_LD="$ld" + return 0 + done + done + + return 1 +} + +out_sections() { $READELF -S -W "$workdir/out.o" 2>/dev/null; } +out_relocs() { $READELF -r -W "$workdir/out.o" 2>/dev/null; } +out_symbols() { $READELF -s -W "$workdir/out.o" 2>/dev/null; } +diff_log() { cat "$workdir/diff.log"; } + +# out_strings
+# +# The strings in one section of the output, for the names livepatch matches on. +out_strings() { $READELF -p "$1" "$workdir/out.o" 2>/dev/null; } + +# Checks on the input objects, to run before klp diff. The two forms differ in +# what an absent construct means: +# +# require_* the compiler cannot produce it here -> skip +# assert_* the fixture is supposed to produce it -> fail + +in_sections() { $READELF -S -W "$workdir/$1" 2>/dev/null; } +in_symbols() { $READELF -s -W "$workdir/$1" 2>/dev/null; } +in_relocs() { $READELF -r -W "$workdir/$1" 2>/dev/null; } + +# count_input_symbols +# +# How many object symbols of exactly that name the input has. Deliberately not +# a grep: readelf lists section symbols too, and a newer binutils prints their +# name -- ".data." -- where an older one leaves the column blank. A dot +# is not a word character, so "grep -w " counts that line as well, and +# the same object gives a different answer depending on which readelf reads it. +count_input_symbols() +{ + in_symbols "$1" | awk -v n="$2" '$4 == "OBJECT" && $8 == n' | wc -l +} + +# re_quote +# +# A string as a literal basic regular expression. Nearly every name these +# assertions match on contains a dot -- .text.target, .klp.rela.vmlinux -- and +# an unescaped dot matches any character, so an assertion for one section can be +# satisfied by a different one whose name merely lines up. +re_quote() { printf '%s' "$1" | sed 's|[].[^$*\\/]|\\&|g'; } + +has_input_section() { in_sections "$1" | grep -q "[[:space:]]$(re_quote "$2")[[:space:]]"; } +has_input_symbol() { in_symbols "$1" | awk -v n="$2" '$NF == n' | grep -q .; } + +assert_input_section() +{ + local obj + + for obj in "$orig_obj" "$patched_obj"; do + has_input_section "$obj" "$1" || + fail "fixture produced no section '$1' in $obj" + done +} + +assert_input_symbol() +{ + local obj + + for obj in "$orig_obj" "$patched_obj"; do + has_input_symbol "$obj" "$1" || + fail "fixture produced no symbol '$1' in $obj" + done +} + +require_input_section() +{ + local obj + + for obj in "$orig_obj" "$patched_obj"; do + has_input_section "$obj" "$1" || + probe_skip "compiler produced no section '$1' here" + done +} + +assert_section() +{ + out_sections | grep -q "[[:space:]]$(re_quote "$1")[[:space:]]" || + fail "expected section '$1' in output" +} + +assert_no_section() +{ + out_sections | grep -q "[[:space:]]$(re_quote "$1")[[:space:]]" && + fail "unexpected section '$1' in output" + return 0 +} + +assert_patched() +{ + assert_section ".text.$1" +} + +assert_not_patched() +{ + out_sections | grep -q "[[:space:]]$(re_quote ".text.$1")[[:space:]]" && + fail "function '$1' should not have been cloned" + return 0 +} + +# section_relocs
+# +# The relocations against one section. readelf prints every relocation section +# in turn, so a test asking about ".smp_locks" has to cut its block out of the +# listing first. +section_relocs() +{ + local sec="${1//./\\.}" + + out_relocs | awk "/rela$sec'/,/^\$/" +} + +assert_reloc_sym() +{ + section_relocs "$1" | awk -v n="$2" '$5 == n' | grep -q . || + fail "expected a relocation to '$2' in '$1'" +} + +assert_no_reloc_sym() +{ + section_relocs "$1" | awk -v n="$2" '$5 == n' | grep -q . && + fail "unexpected relocation to '$2' in '$1'" + return 0 +} + +# assert_reloc_count
+# +# Counts relocation entries, not header or blank lines: whether a special +# section entry was extracted once, twice or not at all is usually the whole +# question. +# +# A count of zero is ambiguous on its own -- a section with no relocations and +# no section at all both read as zero -- so require the section to exist. A +# test expecting nothing there wants assert_no_section. +assert_reloc_count() +{ + local n + + assert_section "$1" + + n="$(section_relocs "$1" | grep -cE '^[0-9a-f]{8,}')" + [ "$n" = "$2" ] || + fail "expected $2 relocations in '$1', found $n" +} + +# assert_klp_sym [object] +# +# A klp symbol is named .klp.sym..,. The object +# defaults to any, since most tests care that the reference was converted at +# all rather than which object it resolved against. +assert_klp_sym() +{ + out_symbols | grep -q "\.klp\.sym\.${2:-[^.]*}\.$(re_quote "$1")," || + fail "expected klp symbol for '$1'" +} + +# assert_klp_sympos +# +# The number after the comma in .klp.sym.., says which +# of several same-named symbols livepatch should resolve to, counting from 1; +# 0 means the name is unique and no disambiguation is needed. Resolving to the +# wrong one is not a load failure, it is a patch quietly wired to the wrong +# object. +assert_klp_sympos() +{ + out_symbols | grep -qE "\.klp\.sym\.[^.]+\.$(re_quote "$1"),$2([[:space:]]|\$)" || + fail "expected klp symbol for '$1' with sympos $2, found:$( + out_symbols | grep -o "\.klp\.sym\.[^.]*\.$(re_quote "$1"),[0-9]*" | + sort -u | tr '\n' ' ')" +} + +assert_no_klp_sym() +{ + out_symbols | grep -q "\.klp\.sym\.${2:-[^.]*}\.$(re_quote "$1")," && + fail "unexpected klp symbol for '$1'" + return 0 +} + +assert_tombstone() +{ + out_symbols | grep -qE "\.klp\.tombstone\.$(re_quote "$1")([[:space:]]|\$)" || + fail "expected a tombstone for '$1'" +} + +assert_symbol() +{ + out_symbols | awk -v n="$1" '$NF == n' | grep -q . || + fail "expected symbol '$1' in output" +} + +assert_no_symbol() +{ + out_symbols | awk -v n="$1" '$NF == n' | grep -q . && + fail "unexpected symbol '$1' in output" + return 0 +} + +# assert_diff_log +# +# klp diff's combined output, for tests asserting on a diagnostic. Error +# messages are part of the interface when the whole point is that a construct +# gets rejected, and a rejection for the wrong reason is not a pass. +assert_diff_log() +{ + diff_log | grep -qE -- "$1" || + fail "expected '$1' in klp diff output: $(tail -2 "$workdir/diff.log")" +} + +# checksum_of +# +# The checksum "klp checksum" recorded for one symbol, as a hex string. +# +# .discard.sym_checksum is an array of { u64 addr; u64 checksum; }, where addr +# is the target of a relocation naming the symbol. Nothing in the section +# itself says which symbol an entry belongs to, so the relocation is what +# locates the entry; the checksum is the eight bytes after it. +# Callers use this in a command substitution, where fail() would only exit the +# subshell and the test would carry on with an empty checksum. So this returns +# non-zero and prints nothing, and the assertions below check for that. For +# the same reason it does not run run_checksum() itself: that one does call +# fail(), and from in here the message would be captured as the checksum +# rather than ending the test. The caller runs it first. +checksum_of() +{ + local obj="$workdir/$1" sym="$2" off + + off="$($READELF -rW "$obj" 2>/dev/null | + awk -v s="$sym" '/rela\.discard\.sym_checksum/,/^$/ { + if ($5 == s) { print $1; exit } + }')" + + [ -n "$off" ] || return 1 + + $OBJCOPY -O binary --only-section=.discard.sym_checksum \ + "$obj" "$workdir/checksums.bin" 2>/dev/null || return 1 + + dd if="$workdir/checksums.bin" bs=1 skip=$((16#$off + 8)) count=8 \ + status=none | od -An -tx1 | tr -d ' \n' +} + +# assert_checksum_differs / assert_checksum_matches +# +# Compare what klp checksum recorded for a symbol in the original against the +# patched object. This is what decides whether klp diff treats a function as +# changed, so a test asserting only that the right functions were cloned cannot +# tell a correct checksum from one which happens to differ. +checksum_pair() +{ + run_checksum + + orig_checksum="$(checksum_of "$orig_obj" "$1")" + patched_checksum="$(checksum_of "$patched_obj" "$1")" + + [ -n "$orig_checksum" ] || + fail "no checksum recorded for '$1' in $orig_obj" + [ -n "$patched_checksum" ] || + fail "no checksum recorded for '$1' in $patched_obj" +} + +assert_checksum_differs() +{ + checksum_pair "$1" + + [ "$orig_checksum" != "$patched_checksum" ] || + fail "checksum for '$1' unchanged at $orig_checksum, expected it to differ" +} + +assert_checksum_matches() +{ + checksum_pair "$1" + + [ "$orig_checksum" = "$patched_checksum" ] || + fail "checksum for '$1' changed from $orig_checksum to" \ + "$patched_checksum, expected no change" +} + +# run_post_link [expected exit status] +# +# klp post-link runs last in a livepatch build, converting the intermediate +# __klp_relocs.* sections into the .klp.rela.* form the kernel consumes. It +# needs nothing but an object containing those sections, which is what klp diff +# produces, so it runs on out.o here rather than on a built module. Rewrites +# out.o in place, so the out_* helpers show the result afterwards. +run_post_link() +{ + local expect="${1:-0}" rc=0 + + "$OBJTOOL" klp post-link "$workdir/out.o" \ + > "$workdir/post-link.log" 2>&1 || rc=$? + + [ "$rc" = "$expect" ] || + fail "klp post-link exited $rc, expected $expect:" \ + "$(tail -2 "$workdir/post-link.log")" +} + +# The flags readelf prints for a section, or nothing when it has none. The +# leading "[nn]" index is stripped first so the columns can be counted. +section_flags() +{ + out_sections | sed 's/^ *\[[ 0-9]*\] *//' | + awk -v s="$1" '$1 == s && $7 ~ /^[A-Za-z]+$/ { print $7 }' +} + +# assert_section_flag
+# +# SHF_RELA_LIVEPATCH is OS-specific, so readelf renders it as "o". A klp rela +# section which lost it is an ordinary rela section, which the linker may apply +# and the livepatch code will not. +assert_section_flag() +{ + local flags; flags="$(section_flags "$1")" + + [ -n "$flags" ] || + fail "section '$1' has no flags, expected '$2'" + case "$flags" in + *"$2"*) ;; + *) fail "section '$1' has flags '$flags', expected '$2'" ;; + esac +} + +# assert_klp_rela
+# +# post-link names the converted sections .klp.rela..
, one per +# base section. Also checks SHF_RELA_LIVEPATCH, since the name alone is not +# what makes the kernel process it. +assert_klp_rela() +{ + local name=".klp.rela.$1.$2" + + out_sections | grep -q "[[:space:]]$(re_quote "$name")[[:space:]]" || + fail "expected section '$name' in output" + + assert_section_flag "$name" o +} + +# assert_livepatch_sym +# +# Symbols a klp relocation resolves against live in SHN_LIVEPATCH, which +# readelf prints as "OS [0xff20]" -- llvm-readelf without the space, so match +# either. The kernel resolves these itself at patch load; anything else is a +# symbol the module loader will try, and fail, to resolve normally. +assert_livepatch_sym() +{ + out_symbols | grep -E 'OS ?\[0xff20\]' | + grep -qE "\.klp\.sym\.[^.]+\.$(re_quote "$1")," || + fail "expected a klp symbol for '$1' in SHN_LIVEPATCH" +} diff --git a/tools/objtool/tests/run-tests.sh b/tools/objtool/tests/run-tests.sh new file mode 100755 index 00000000000000..c0762f532d2a3c --- /dev/null +++ b/tools/objtool/tests/run-tests.sh @@ -0,0 +1,239 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Run the objtool klp tests. Each test-*.sh prints one TAP result line. +# +# Tests live in generic/ and in a directory per architecture. A run executes +# generic/ plus the one matching this architecture, so a test which cannot +# apply here is not run rather than reporting a skip; what was left out is +# reported once, as a comment, so differing coverage is still visible. +# +# A run covers one compiler and one architecture; CI runs the combinations. +# The harness checks the environment once up front and fails the run if the +# suite cannot execute, rather than letting every test skip and exit 0. + +set -u + +# Determinism: the order test-*.sh expands in, and how grep's character ranges +# and sort's collation behave inside the tests, are all locale-dependent. A +# suite whose results depend on the invoking shell's locale is a suite whose +# failures cannot be reproduced. +export LC_ALL=C + +usage() +{ + cat <&2; usage 1 ;; + *) break ;; + esac +done + +KLP_TEST_KEEP="${KEEP:-failed}" +[ -n "$keep_from_args" ] && KLP_TEST_KEEP="$keep_from_args" +case "$KLP_TEST_KEEP" in +all|none|failed) ;; +*) + echo "invalid KEEP=$KLP_TEST_KEEP (want failed, all, or none)" >&2 + exit 1 + ;; +esac +export KLP_TEST_KEEP + +echo "TAP version 13" + +# Sourcing the harness runs its preflight, which decides which architecture +# this run is for -- so the test list cannot be built before it has, and the +# tests inherit the answers rather than working them out again. +. ./lib.sh + +dirs=( generic ) +[ -d "$KLP_TEST_ARCH" ] && dirs+=( "$KLP_TEST_ARCH" ) + +if [ $# -gt 0 ]; then + tests=() + for arg in "$@"; do + name="test-${arg#test-}"; name="${name%.sh}.sh" + found= + for d in "${dirs[@]}"; do + [ -f "$d/$name" ] || continue + [ -x "$d/$name" ] || + { echo "not executable: $d/$name" >&2; exit 1; } + tests+=( "$d/$name" ); found=y + done + [ -n "$found" ] || + { echo "no such test for $KLP_TEST_ARCH: $arg" >&2; exit 1; } + done +else + tests=() + for d in "${dirs[@]}"; do + for t in "$d"/test-*.sh; do + [ -f "$t" ] && tests+=( "$t" ) + done + done + [ "${#tests[@]}" -gt 0 ] || + { echo "1..0 # SKIP no tests found"; exit 0; } + + # Tests for another architecture are absent from this run entirely. Say + # how many, so a run which covers less than the tree holds does not look + # like one that covers all of it. + for d in */; do + d="${d%/}" + case "$d" in generic|"$KLP_TEST_ARCH") continue ;; esac + n=$(ls "$d"/test-*.sh 2>/dev/null | wc -l) + [ "$n" -gt 0 ] || continue + echo "# not run: $n test$( [ "$n" = 1 ] || echo s ) in $d/" \ + "(this run is $KLP_TEST_ARCH)" + done +fi + +# One directory for the whole run, one per test inside it, mirroring the +# source layout. A run then leaves a single thing behind instead of 39 +# scattered among everything else using mktemp. +rundir="$(mktemp -d "${TMPDIR:-/tmp}/klp-tests.XXXXXXXX")" || + { echo "Bail out! cannot create a working directory" >&2; exit 1; } + +# The run's own two levels: each test's directory, and the one per source +# directory holding them. Take them away if the tests left them empty, and +# say so if they did not. Either rmdir may fail -- the glob stays unexpanded +# when nothing was created -- so ask the directory itself rather than trusting +# the status. Never rm -rf: what to keep is the tests' decision, made in +# cleanup() as each one exits, and this must not overrule it. +reap_rundir() +{ + rmdir "$rundir"/*/ 2>/dev/null + rmdir "$rundir" 2>/dev/null + [ -d "$rundir" ] +} + +# An interrupted run has the same directory to answer for, and the tests it +# never reached will not clean up on their way out. The one it was running +# has, and under the default it kept what it had built, so say where. +interrupted() +{ + reap_rundir && echo "# interrupted; what was built is in $rundir" + exit 130 +} +trap interrupted INT TERM HUP + +echo "1..${#tests[@]}" + +pass=0 fail=0 static_skip=0 probe_skip=0 xfail=0 xpass=0 +failed_dirs=() + +for t in "${tests[@]}"; do + out="$(KLP_TEST_WORKDIR="$rundir/${t%.sh}" ./"$t" 2>&1)" + rc=$? + + # A test prints one result line, but it is not necessarily the only + # thing it prints: objtool warns on stderr, and the runner captures + # that. Classify the result line itself rather than the whole of the + # output, or a stray line ahead of it makes every pattern below miss and + # the exit status decide -- which would count an expected failure, which + # exits 0, as a pass. + result="$(printf '%s\n' "$out" | grep -E '^(ok|not ok)' | tail -1)" + rest="$(printf '%s\n' "$out" | grep -Ev '^(ok|not ok)')" + + # Classify from the result line, not the exit status: a skip and a pass + # both exit 0, and telling them apart is the point of counting. + # + # The two skip kinds differ in what they promise. A static skip was + # declared before the test ran ("clang does not do this"), so it is + # expected indefinitely. A probe skip means the construct did not turn + # up this time, which is weaker and worth watching: one that becomes + # permanent is a fixture that quietly stopped testing anything. + case "$result" in + *"# SKIP (declared)"*) static_skip=$((static_skip + 1)) ;; + *"# SKIP (probe)"*) probe_skip=$((probe_skip + 1)) ;; + *"# SKIP"*) + # An undeclared skip: the test gave up for a reason it never + # said it might. That is a hole, not an expected outcome. + # + # Replace the line rather than adding one. Every test owes the + # plan exactly one result, and a consumer counting them is + # entitled to say so when the totals disagree. + rest="$rest${rest:+$'\n'}was: $result" + result="not ok - $(basename "$t" .sh): undeclared skip" + result="$result (use gcc_only/clang_only or require_input_*)" + fail=$((fail + 1)); failed_dirs+=( "$rundir/${t%.sh}" ) ;; + "not ok"*"# TODO"*) xfail=$((xfail + 1)) ;; + "ok"*"# TODO"*) xpass=$((xpass + 1)); failed_dirs+=( "$rundir/${t%.sh}" ) ;; + "not ok"*) fail=$((fail + 1)); failed_dirs+=( "$rundir/${t%.sh}" ) ;; + "ok"*) pass=$((pass + 1)) ;; + *) + # No result line at all: the test died before reporting. + rest="$rest${rest:+$'\n'}exited $rc without a result line" + result="not ok - $(basename "$t" .sh): no TAP result" + fail=$((fail + 1)); failed_dirs+=( "$rundir/${t%.sh}" ) ;; + esac + + echo "$result" + [ -n "$rest" ] && printf '%s\n' "$rest" | sed 's/^[^#]/# &/' + +done + +echo "# pass:$pass fail:$fail static-skip:$static_skip" \ + "probe-skip:$probe_skip xfail:$xfail xpass:$xpass" + +case "$KLP_TEST_KEEP" in +all) + echo "# keep=all: workdirs kept in $rundir" + echo "# inspect: diff.log, readelf -S out.o under each test-* subdirectory" + echo "# cleanup: rm -rf $rundir" + ;; +failed) + if [ "${#failed_dirs[@]}" -gt 0 ]; then + echo "# keep=failed: ${#failed_dirs[@]} failing test(s) kept under $rundir:" + for d in "${failed_dirs[@]}"; do + echo "# ${d#"$rundir"/}/" + done + echo "# inspect: diff.log readelf -S out.o" + echo "# one test: $PWD/run-tests.sh " + echo "# cleanup: rm -rf $rundir" + elif reap_rundir; then + echo "# $rundir was not empty;" \ + "a test did not clean up after itself" + fi + ;; +none) + if reap_rundir; then + echo "# $rundir was not empty;" \ + "a test did not clean up after itself" + elif [ "$fail" != 0 ] || [ "$xpass" != 0 ]; then + echo "# keep=none: artifacts were removed" \ + "(re-run with KEEP=failed or KEEP=all)" + fi + ;; +esac + +[ "$fail" = 0 ] && [ "$xpass" = 0 ] diff --git a/tools/objtool/tests/x86/fixtures/alt_annotate.c b/tools/objtool/tests/x86/fixtures/alt_annotate.c new file mode 100644 index 00000000000000..af44d320dcb05b --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/alt_annotate.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * An x86 alternative whose replacement instruction carries a text annotation. + * + * The kernel does this wherever an ALTERNATIVE contains something objtool has + * to be told about -- a retpoline-safe indirect branch, an intentionally + * missing ENDBR -- so the .discard.annotate_insn entry references an address + * inside .altinstr_replacement rather than inside a function. + * + * Two things make that awkward for klp diff, and both are why this fixture + * exists. Replacement code has no real symbol: objtool invents a NOTYPE fake + * symbol for it, so an annotation pointing there does not reference a FUNC. + * And .discard.annotate_insn has to be cloned after .altinstructions, or the + * replacement it names has no clone to point at yet. + * + * struct alt_instr is written out by hand as in empty_alternative.c: s32 + * instr_offset, s32 repl_offset, u32 ft_flags, u8 instrlen, u8 replacementlen, + * with an entsize so klp diff can find the entry boundaries. + * .discard.annotate_insn entries are s32 offset, s32 type; type 2 is + * ANNOTYPE_RETPOLINE_SAFE. + * + * The replacement label is global so the relocations name it rather than + * .altinstr_replacement plus an addend, which klp diff cannot convert. It is + * still NOTYPE, which is the shape that matters here. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +int target(int x) +{ + asm volatile( + "661: nop\n\t" + ".pushsection .altinstr_replacement, \"ax\"\n\t" + ".globl target_repl\n\t" + "target_repl:\n\t" + " nop\n\t" + /* The annotation lands inside the replacement. */ + ".pushsection .discard.annotate_insn, \"M\", @progbits, 8\n\t" + ".long target_repl - .\n\t" + ".long 2\n\t" + ".popsection\n\t" + "target_repl_end:\n\t" + ".popsection\n\t" + ".pushsection .altinstructions, \"aM\", @progbits, 14\n\t" + ".long 661b - .\n\t" + ".long target_repl - .\n\t" + ".long 0\n\t" + ".byte 1\n\t" + ".byte target_repl_end - target_repl\n\t" + ".popsection\n\t"); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/x86/fixtures/checksum_alt.c b/tools/objtool/tests/x86/fixtures/checksum_alt.c new file mode 100644 index 00000000000000..ed342d94eb9ed3 --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/checksum_alt.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * An x86 alternative whose replacement code is part of the patched function's + * checksum. + * + * checksum_update_insn() walks insn->alts after hashing the instruction + * itself, hashing the alternative's type and, when the replacement forms a + * group, its feature number and every instruction in it. So editing only the + * replacement -- code the CPU may or may not ever run -- has to move the + * function's checksum. + * + * It is reached through objtool's own alternative handling, so the object has + * to go through the check pass first: insn->alts is built there, not by the + * compiler. + * + * struct alt_instr is written out by hand as in empty_alternative.c: s32 + * instr_offset, s32 repl_offset, u32 ft_flags, u8 instrlen, u8 replacementlen. + * + * Variants, applied to the patched build only: + * + * ALT_REPL the replacement instruction changes; the original does not + * ALT_FEATURE the feature number changes; no code changes at all + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* + * Both spellings are two bytes, because a replacement may not be longer than + * the instruction it replaces: "xchg %ax, %ax" is 66 90 and two nops are + * 90 90. The original below is padded to match. + */ +#if defined(PATCHED) && defined(ALT_REPL) +#define REPL_INSN " nop\n\t nop\n\t" +#else +#define REPL_INSN " xchg %ax, %ax\n\t" +#endif + +#if defined(PATCHED) && defined(ALT_FEATURE) +#define FEATURE "7" +#else +#define FEATURE "3" +#endif + +int target(int x) +{ + asm volatile( + "661: nop\n\t" + " nop\n\t" + "662:\n\t" + ".pushsection .altinstr_replacement, \"ax\"\n\t" + ".globl target_repl\n\t" + "target_repl:\n\t" + REPL_INSN + "target_repl_end:\n\t" + ".popsection\n\t" + ".pushsection .altinstructions, \"aM\", @progbits, 14\n\t" + ".long 661b - .\n\t" + ".long target_repl - .\n\t" + ".long " FEATURE "\n\t" + ".byte 662b - 661b\n\t" + ".byte target_repl_end - target_repl\n\t" + ".popsection\n\t"); + + return x + 1; +} diff --git a/tools/objtool/tests/x86/fixtures/empty_alternative.c b/tools/objtool/tests/x86/fixtures/empty_alternative.c new file mode 100644 index 00000000000000..9336d74bfa9265 --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/empty_alternative.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * An x86 alternative with an empty replacement, as the second entry of + * ALTERNATIVE_2("orig", "repl", ft1, "", ft2) produces. Its replacement + * offset still gets a relocation, but the label it points at is the end of the + * previous replacement, which is also where the *next* one begins -- here, + * neighbor()'s. The value is meaningless; it is only ever used with a length + * of zero. + * + * struct alt_instr is written out by hand so the fixture builds without kernel + * headers: s32 instr_offset, s32 repl_offset, u32 ft_flags, u8 instrlen, + * u8 replacementlen. The section carries an entsize because klp diff needs + * either that or an ANNOTATE_DATA_SPECIAL annotation to find entry boundaries. + * + * The replacement labels are global so the relocations name them rather than + * .altinstr_replacement plus an addend. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +extern int neighbor_only(int x); + +int target(int x) +{ + asm volatile( + "661: nop\n\t" + ".pushsection .altinstr_replacement, \"ax\"\n\t" + ".globl target_repl\n\t" + "target_repl:\n\t" + " nop\n\t" + "target_repl_end:\n\t" + ".popsection\n\t" + ".pushsection .altinstructions, \"aM\", @progbits, 14\n\t" + /* a real replacement */ + ".long 661b - .\n\t" + ".long target_repl - .\n\t" + ".long 0\n\t" + ".byte 1\n\t" + ".byte target_repl_end - target_repl\n\t" + /* an empty one, pointing at neighbor()'s replacement */ + ".long 661b - .\n\t" + ".long neighbor_repl - .\n\t" + ".long 0\n\t" + ".byte 1\n\t" + ".byte 0\n\t" + ".popsection\n\t"); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} + +/* + * Unrelated, unpatched, and referencing a symbol nothing else does, so that + * dragging its replacement in is visible. + */ +int neighbor(int x) +{ + asm volatile( + "771: nop\n\t" + ".pushsection .altinstr_replacement, \"ax\"\n\t" + ".globl neighbor_repl\n\t" + "neighbor_repl:\n\t" + " call neighbor_only\n\t" + "neighbor_repl_end:\n\t" + ".popsection\n\t" + ".pushsection .altinstructions, \"aM\", @progbits, 14\n\t" + ".long 771b - .\n\t" + ".long neighbor_repl - .\n\t" + ".long 0\n\t" + ".byte 1\n\t" + ".byte neighbor_repl_end - neighbor_repl\n\t" + ".popsection\n\t"); + return x; +} diff --git a/tools/objtool/tests/x86/fixtures/kcfi.c b/tools/objtool/tests/x86/fixtures/kcfi.c new file mode 100644 index 00000000000000..b62fc60634cf22 --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/kcfi.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * An indirect call, which under kCFI is preceded by a type check and a trap. + * + * Clang emits a __cfi_ prefix symbol carrying the type hash ahead of + * every address-taken function, and records the trap site in .kcfi_traps. + * Both belong to the function and both have to come with it into a patch. + * + * Needs -fsanitize=kcfi, which only Clang has. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +static int impl_a(int x) +{ + return x + 1; +} + +static int impl_b(int x) +{ + return x * 2; +} + +__attribute__((noinline)) int (*pick(int x))(int) +{ + return (x & 1) ? impl_a : impl_b; +} + +int target(int x) +{ + int (*fn)(int arg) = pick(x); + +#ifdef PATCHED + return fn(x) + 2; +#else + return fn(x) + 1; +#endif +} diff --git a/tools/objtool/tests/x86/fixtures/special_sections.c b/tools/objtool/tests/x86/fixtures/special_sections.c new file mode 100644 index 00000000000000..d42798848fda04 --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/special_sections.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A special section entry belonging to a patched function. SPECIAL_SEC picks + * which section, since klp diff treats eight of them alike and each needs + * extracting for the patched function and no other. + * + * The entry is written out by hand so the fixture builds without kernel + * headers. Only the leading relocation matters to klp diff; the rest is + * padded to the section's real entry size, because the entries have to be the + * right length for the boundaries between them to fall in the right places. + * + * SPECIAL_RELOCS covers __ex_table, whose entries relocate both the faulting + * instruction and its fixup; objtool rejects one with only the first. + */ + +#ifndef SPECIAL_SEC +#define SPECIAL_SEC "__bug_table" +#endif +#ifndef SPECIAL_ENTSIZE +#define SPECIAL_ENTSIZE 12 +#endif +#ifndef SPECIAL_RELOCS +#define SPECIAL_RELOCS 1 +#endif + +#define STR_(x) #x +#define STR(x) STR_(x) + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux"; + +/* + * other() gets an entry of its own, so that "the untouched function's entry + * was not dragged in" is a question the section can actually answer. With + * only target() contributing, there is nothing for klp diff to leave behind + * and the negative assertion holds however the extraction behaves. + */ +int other(int x) +{ + asm volatile( + "3: nop\n\t" + "4:\n\t" + ".pushsection " SPECIAL_SEC ", \"aM\", @progbits, " + STR(SPECIAL_ENTSIZE) "\n\t" + ".long 3b - .\n\t" +#if SPECIAL_RELOCS > 1 + ".long 4b - .\n\t" + ".fill " STR(SPECIAL_ENTSIZE) " - 8, 1, 0\n\t" +#else + ".fill " STR(SPECIAL_ENTSIZE) " - 4, 1, 0\n\t" +#endif + ".popsection\n\t"); + + return x + 9; +} + +int target(int x) +{ + asm volatile( + "1: nop\n\t" + "2:\n\t" + ".pushsection " SPECIAL_SEC ", \"aM\", @progbits, " + STR(SPECIAL_ENTSIZE) "\n\t" + ".long 1b - .\n\t" +#if SPECIAL_RELOCS > 1 + ".long 2b - .\n\t" + ".fill " STR(SPECIAL_ENTSIZE) " - 8, 1, 0\n\t" +#else + ".fill " STR(SPECIAL_ENTSIZE) " - 4, 1, 0\n\t" +#endif + ".popsection\n\t"); +#ifdef PATCHED + return x + 2; +#else + return x + 1; +#endif +} diff --git a/tools/objtool/tests/x86/fixtures/static_call_no_key.c b/tools/objtool/tests/x86/fixtures/static_call_no_key.c new file mode 100644 index 00000000000000..748ba7c0f86dff --- /dev/null +++ b/tools/objtool/tests/x86/fixtures/static_call_no_key.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * A static call to a trampoline whose key symbol this object cannot see. + * + * That is the normal situation for a module: __SCK__* keys are not exported, + * and read-only access is granted at load time instead. objtool's static call + * handling has to accept it for any module, including a livepatch module built + * by hand rather than by klp-build. + * + * LIVEPATCH adds the .modinfo tag which makes objtool treat this as a + * livepatch module. + */ + +static const char __modinfo[] + __attribute__((section(".modinfo"), used, aligned(1))) = +#ifdef LIVEPATCH + "\0livepatch=Y" +#endif + "\0name=klp_testmod"; + +/* + * The trampoline is undefined here, exactly as it is for a module calling a + * static call defined in vmlinux. No __SCK__klp_test_call accompanies it. + */ +extern void __SCT__klp_test_call(void); + +int target(int x) +{ + __asm__ volatile("call __SCT__klp_test_call\n\t" ::: "memory"); + + return x + 1; +} diff --git a/tools/objtool/tests/x86/test-alt-annotation.sh b/tools/objtool/tests/x86/test-alt-annotation.sh new file mode 100755 index 00000000000000..96760e6df9e5f5 --- /dev/null +++ b/tools/objtool/tests/x86/test-alt-annotation.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A text annotation on an instruction inside an alternative's replacement must +# be carried into the patch. +# +# The kernel annotates replacement code wherever objtool has to be told +# something about it -- a retpoline-safe indirect branch, a deliberately absent +# ENDBR. Two things made klp diff drop those annotations: +# +# - replacement code has no real symbol, so objtool invents a NOTYPE fake +# one, and the extraction only kept references to FUNC symbols; +# - .discard.annotate_insn was processed before .altinstructions, so the +# replacement it referenced had no clone to point at yet. +# +# Nothing fails at build time when the annotation goes missing. It surfaces +# later as objtool warning about, or rejecting, the patched code it was there +# to explain. +# +# Fixed by 62a7a01fde87 ("objtool/klp: Fix extraction of text annotations for +# alternatives"). + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair alt_annotate.c + +assert_input_section .altinstructions +assert_input_section .discard.annotate_insn + +run_diff + +# The annotation has to survive, and to still name the replacement. Checking +# only the section would pass on an entry whose relocation was dropped. +assert_section .discard.annotate_insn +assert_reloc_sym .discard.annotate_insn target_repl + +pass "text annotation on an alternative replacement carried into the patch" diff --git a/tools/objtool/tests/x86/test-checksum-alt.sh b/tools/objtool/tests/x86/test-checksum-alt.sh new file mode 100755 index 00000000000000..74ebfca2b3ff3b --- /dev/null +++ b/tools/objtool/tests/x86/test-checksum-alt.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# An alternative's replacement code counts towards the checksum of the function +# it belongs to. +# +# checksum_update_insn() walks insn->alts after hashing the instruction itself: +# the alternative's type, and where the replacement forms a group, its feature +# number and every instruction in it. So a patch which edits only the +# replacement -- code that runs on some CPUs and not others -- still has to +# move the function's checksum. +# +# If it does not, klp diff decides the function is unchanged and leaves it out. +# The patch then ships the old replacement, and the bug is fixed only on +# machines whose CPU takes the other arm. Which machines those are depends on +# the feature bit, so the failure looks like a machine-specific bug rather than +# a missing patch. +# +# insn->alts exists only after objtool's check pass, so the pair goes through +# that first -- the compiler emits none of this structure itself. +# +# Covers the same ground as corpus/x86_64/checksum-alt-group, +# checksum-alt-no-group and checksum-alt-recursion-guard in Joe Lawrence's +# klp-build unit test corpus. + +. "$(dirname "$0")/../lib.sh" + +setup + +check() +{ + build_pair checksum_alt.c "-D$1" + assert_input_section .altinstructions + run_objtool_check --mcount + run_checksum + + assert_checksum_differs target +} + +# The replacement instruction itself. +check ALT_REPL +# The feature number, with no instruction anywhere changed. +check ALT_FEATURE + +pass "alternative replacement code counts towards the checksum" diff --git a/tools/objtool/tests/x86/test-empty-alternative.sh b/tools/objtool/tests/x86/test-empty-alternative.sh new file mode 100755 index 00000000000000..9d40c3a405af0c --- /dev/null +++ b/tools/objtool/tests/x86/test-empty-alternative.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# An x86 alternative with an empty replacement still gets a relocation for its +# replacement offset, but the label it points at is the end of the previous +# replacement -- which is also the start of the next one. The value is +# meaningless, and get_alt_entry() already ignores it. +# +# Cloning it drags in an unrelated neighboring replacement and everything that +# replacement references. In the reported case an empty alternative in +# meminfo_proc_show() pulled in one from proc_kcore_init(), emitting a klp +# relocation against init text which is long freed by the time the patch is +# applied. + +. "$(dirname "$0")/../lib.sh" + +setup +build_pair empty_alternative.c + +assert_input_section .altinstructions +assert_input_section .altinstr_replacement + +run_diff + +# target's own replacement comes along ... +assert_symbol target_repl +# ... neighbor's does not, nor what it references. +assert_no_symbol neighbor_repl +assert_no_symbol neighbor_only + +pass "empty alternative's replacement offset ignored when cloning" diff --git a/tools/objtool/tests/x86/test-kcfi.sh b/tools/objtool/tests/x86/test-kcfi.sh new file mode 100755 index 00000000000000..b583ef6087472f --- /dev/null +++ b/tools/objtool/tests/x86/test-kcfi.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Under kCFI an indirect call checks a type hash before jumping, and traps on a +# mismatch. Two things belong to the calling function and must come with it +# into a patch: +# +# - the __cfi_ prefix symbol holding the hash. Lose it and the patched +# function has no type identity, so indirect calls to it trap. +# - its .kcfi_traps entry. Lose that and the trap is not recognised as a +# CFI failure, so what should be a clean report becomes an oops. +# +# Neither shows up at build time. + +. "$(dirname "$0")/../lib.sh" + +clang_only "kCFI is a Clang feature" + +setup + +# Declared above that this is Clang's; a given Clang may still be too old. +cc_supports -fsanitize=kcfi || + probe_skip "this clang does not support -fsanitize=kcfi" + +build_pair kcfi.c -fsanitize=kcfi + +assert_input_section .kcfi_traps +assert_input_symbol __cfi_target + +run_diff + +assert_patched target + +# The prefix symbol comes with its function ... +assert_symbol __cfi_target +# ... and so does the trap entry. +assert_section .kcfi_traps + +pass "kCFI prefix symbol and trap entry carried with the patched function" diff --git a/tools/objtool/tests/x86/test-manual-klp-static-call.sh b/tools/objtool/tests/x86/test-manual-klp-static-call.sh new file mode 100755 index 00000000000000..6c4d275e35491f --- /dev/null +++ b/tools/objtool/tests/x86/test-manual-klp-static-call.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# objtool's static call handling must accept a livepatch module which cannot +# see a static call's key symbol. +# +# __SCK__* keys are not exported; modules get read-only access at load time +# instead. Livepatch modules built by klp-build do have full access to their +# keys, and a check was added on the strength of that -- but a livepatch module +# can also be written by hand, and samples/livepatch is full of them. One of +# those needs a key it cannot see as soon as it does anything that expands to a +# static call, which with CONFIG_MEM_ALLOC_PROFILING_DEBUG includes allocating +# memory: +# +# samples/livepatch/livepatch-shadow-fix1.o: error: objtool: static_call: +# can't find static_call_key symbol: __SCK__WARN_trap +# +# The module built without the livepatch tag is the control: it takes the same +# path and has always been accepted, so a test which only built the livepatch +# one could not tell this fix from the check being removed altogether. +# +# Fixed by f495054bd12e ("objtool/klp: Fix unexported static call key access +# for manually built livepatch modules"). + +. "$(dirname "$0")/../lib.sh" + +setup + +# Not a klp subcommand: this is objtool's ordinary check pass, which is what +# runs over a hand-built livepatch module during a normal kernel build. +for tag in "" -DLIVEPATCH; do + build_one static_call_no_key.c mod.o $tag + + "$OBJTOOL" --module --static-call "$workdir/mod.o" \ + > "$workdir/objtool.log" 2>&1 || + fail "objtool rejected a ${tag:+livepatch }module which cannot" \ + "see its static call key: $(tail -1 "$workdir/objtool.log")" +done + +pass "livepatch module accepted without access to its static call key" diff --git a/tools/objtool/tests/x86/test-special-sections.sh b/tools/objtool/tests/x86/test-special-sections.sh new file mode 100755 index 00000000000000..8dfaa4fc9a360f --- /dev/null +++ b/tools/objtool/tests/x86/test-special-sections.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# klp diff extracts entries from eight special sections. Between them the +# existing tests reach .kcfi_traps, __jump_table, .static_call_sites and +# .altinstructions; __bug_table, __ex_table and __mcount_loc are covered by +# nothing, though the same extraction code serves all of them. +# +# Losing an entry is quiet in every case and wrong in a different way for each: +# a WARN() in patched code that no longer reports where it came from, an +# exception fixup that is simply not there when the faulting instruction traps, +# a function ftrace can no longer see. + +. "$(dirname "$0")/../lib.sh" + + +# section, entry size, relocations per entry +for spec in "__bug_table 12 1" "__ex_table 12 2" "__mcount_loc 8 1"; do + set -- $spec + sec=$1 + + # A fresh workdir per section: run_diff caches its checksums. + setup + build_pair special_sections.c \ + -DSPECIAL_SEC="\"$1\"" -DSPECIAL_ENTSIZE="$2" -DSPECIAL_RELOCS="$3" + + assert_input_section "$sec" + run_diff + + # Extracted, and pointing at the function that was patched. + assert_section "$sec" + assert_reloc_sym "$sec" target + assert_patched target + + # Nothing belonging to the function that was not. + assert_not_patched other + assert_no_reloc_sym "$sec" other + + cleanup +done + +pass "__bug_table, __ex_table and __mcount_loc entries extracted" diff --git a/tools/perf/trace/beauty/include/uapi/linux/sched.h b/tools/perf/trace/beauty/include/uapi/linux/sched.h index 33a4624285cd41..19ffeba894283e 100644 --- a/tools/perf/trace/beauty/include/uapi/linux/sched.h +++ b/tools/perf/trace/beauty/include/uapi/linux/sched.h @@ -53,7 +53,7 @@ */ #define UNSHARE_EMPTY_MNTNS 0x00100000 /* Unshare an empty mount namespace. */ -#ifndef __ASSEMBLY__ +#ifndef __ASSEMBLER__ /** * struct clone_args - arguments for the clone3 syscall * @flags: Flags for the new process as listed above. diff --git a/tools/power/pm-graph/sleepgraph.py b/tools/power/pm-graph/sleepgraph.py index f6d172254829b2..c8bc3bed3f5a67 100755 --- a/tools/power/pm-graph/sleepgraph.py +++ b/tools/power/pm-graph/sleepgraph.py @@ -1448,7 +1448,7 @@ def xtraInfo(self): # Class: DeviceNode # Description: -# A container used to create a device hierachy, with a single root node +# A container used to create a device hierarchy, with a single root node # and a tree of child nodes. Used by Data.deviceTopology() class DeviceNode: def __init__(self, nodename, nodedepth): diff --git a/tools/testing/kunit/qemu_configs/microblazebe.py b/tools/testing/kunit/qemu_configs/microblazebe.py new file mode 100644 index 00000000000000..f37b7af20142df --- /dev/null +++ b/tools/testing/kunit/qemu_configs/microblazebe.py @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: GPL-2.0-only +from ..qemu_config import QemuArchParams + +QEMU_ARCH = QemuArchParams(linux_arch='microblaze', + kconfig=''' +CONFIG_KERNEL_BASE_ADDR=0x90000000 +CONFIG_XILINX_MICROBLAZE0_USE_MSR_INSTR=1 +CONFIG_XILINX_MICROBLAZE0_USE_PCMP_INSTR=1 +CONFIG_XILINX_MICROBLAZE0_USE_BARREL=1 +CONFIG_XILINX_MICROBLAZE0_USE_DIV=0 +CONFIG_XILINX_MICROBLAZE0_USE_HW_MUL=1 +CONFIG_XILINX_MICROBLAZE0_USE_FPU=0 +CONFIG_CPU_BIG_ENDIAN=y +CONFIG_SERIAL_UARTLITE=y +CONFIG_SERIAL_UARTLITE_CONSOLE=y +CONFIG_MB_POWER_OFF_THROUGH_UNALIGNED_PC=y +''', + qemu_arch='microblaze', + kernel_path='arch/microblaze/boot/linux.bin', + kernel_command_line='kunit_shutdown=poweroff', + extra_qemu_params=[ + '-M', 'petalogix-s3adsp1800', + ], +) diff --git a/tools/testing/kunit/qemu_configs/microblazeel.py b/tools/testing/kunit/qemu_configs/microblazeel.py new file mode 100644 index 00000000000000..a8bc572d892ad3 --- /dev/null +++ b/tools/testing/kunit/qemu_configs/microblazeel.py @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: GPL-2.0-only +from ..qemu_config import QemuArchParams + +QEMU_ARCH = QemuArchParams(linux_arch='microblaze', + kconfig=''' +CONFIG_KERNEL_BASE_ADDR=0x50000000 +CONFIG_XILINX_MICROBLAZE0_FAMILY="virtex5" +CONFIG_XILINX_MICROBLAZE0_USE_MSR_INSTR=1 +CONFIG_XILINX_MICROBLAZE0_USE_PCMP_INSTR=1 +CONFIG_XILINX_MICROBLAZE0_USE_BARREL=1 +CONFIG_XILINX_MICROBLAZE0_USE_DIV=1 +CONFIG_XILINX_MICROBLAZE0_USE_HW_MUL=2 +CONFIG_XILINX_MICROBLAZE0_USE_FPU=1 +CONFIG_CPU_LITTLE_ENDIAN=y +CONFIG_SERIAL_8250=y +CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_SERIAL_8250_16550A_VARIANTS=y +CONFIG_SERIAL_OF_PLATFORM=y +CONFIG_MB_POWER_OFF_THROUGH_UNALIGNED_PC=y +''', + qemu_arch='microblaze', + kernel_path='arch/microblaze/boot/linux.bin', + kernel_command_line='kunit_shutdown=poweroff', + extra_qemu_params=[ + '-M', 'petalogix-ml605', + ], +) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index bc4fdb6f533759..7e8d57904baef3 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -34,10 +34,12 @@ TARGETS += exec TARGETS += fchmodat2 TARGETS += filesystems TARGETS += filesystems/binderfs +TARGETS += filesystems/configfs TARGETS += filesystems/epoll TARGETS += filesystems/eventfd TARGETS += filesystems/failfs TARGETS += filesystems/fat +TARGETS += filesystems/file_stressor TARGETS += filesystems/openat2 TARGETS += filesystems/open_tree_ns TARGETS += filesystems/overlayfs diff --git a/tools/testing/selftests/arm64/fp/fp-ptrace.c b/tools/testing/selftests/arm64/fp/fp-ptrace.c index b435837c8c0e87..9b184d78f4fd6b 100644 --- a/tools/testing/selftests/arm64/fp/fp-ptrace.c +++ b/tools/testing/selftests/arm64/fp/fp-ptrace.c @@ -115,23 +115,6 @@ static void handle_alarm(int sig, siginfo_t *info, void *context) got_alarm = true; } -#ifdef CONFIG_CPU_BIG_ENDIAN -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - u64 a = swab64(x); - u64 b = swab64(x >> 64); - - return ((__uint128_t)a << 64) | b; -} -#else -static __uint128_t arm64_cpu_to_le128(__uint128_t x) -{ - return x; -} -#endif - -#define arm64_le128_to_cpu(x) arm64_cpu_to_le128(x) - static bool sve_supported(void) { return getauxval(AT_HWCAP) & HWCAP_SVE; @@ -932,7 +915,7 @@ static void fpsimd_to_sve(__uint128_t *v, char *z, int vl) for (i = 0; i < __SVE_NUM_ZREGS; i++) { p = (__uint128_t *)&z[__SVE_ZREG_OFFSET(vq, i)]; - *p = arm64_cpu_to_le128(v[i]); + *p = v[i]; } } diff --git a/tools/testing/selftests/arm64/gcs/gcs-locking.c b/tools/testing/selftests/arm64/gcs/gcs-locking.c index 1e6abb136ffd07..80a1317f6097b0 100644 --- a/tools/testing/selftests/arm64/gcs/gcs-locking.c +++ b/tools/testing/selftests/arm64/gcs/gcs-locking.c @@ -77,6 +77,21 @@ FIXTURE_VARIANT_ADD(valid_modes, enable_write_push) PR_SHADOW_STACK_PUSH, }; +FIXTURE_VARIANT_ADD(valid_modes, write) +{ + .mode = PR_SHADOW_STACK_WRITE, +}; + +FIXTURE_VARIANT_ADD(valid_modes, push) +{ + .mode = PR_SHADOW_STACK_PUSH, +}; + +FIXTURE_VARIANT_ADD(valid_modes, write_push) +{ + .mode = PR_SHADOW_STACK_WRITE | PR_SHADOW_STACK_PUSH, +}; + FIXTURE_SETUP(valid_modes) { } @@ -115,7 +130,7 @@ TEST_F(valid_modes, enable_lock_disable) ASSERT_EQ(ret, 0); ret = my_syscall2(__NR_prctl, PR_SET_SHADOW_STACK_STATUS, 0); - ASSERT_EQ(ret, -EBUSY); + ASSERT_EQ(ret, -EPERM); _exit(0); } @@ -131,7 +146,7 @@ TEST_F(valid_modes, lock_enable) ret = my_syscall2(__NR_prctl, PR_SET_SHADOW_STACK_STATUS, variant->mode); - ASSERT_EQ(ret, -EBUSY); + ASSERT_EQ(ret, -EPERM); ret = prctl(PR_GET_SHADOW_STACK_STATUS, &mode, 0, 0, 0); ASSERT_EQ(ret, 0); diff --git a/tools/testing/selftests/arm64/mte/check_buffer_fill.c b/tools/testing/selftests/arm64/mte/check_buffer_fill.c index 039b1d7d856637..54e1dcaf52ac2f 100644 --- a/tools/testing/selftests/arm64/mte/check_buffer_fill.c +++ b/tools/testing/selftests/arm64/mte/check_buffer_fill.c @@ -41,7 +41,7 @@ static int check_buffer_by_byte(int mem_type, int mode) mte_initialize_current_context(mode, (uintptr_t)ptr, sizes[i]); /* Set some value in tagged memory */ for (j = 0; j < sizes[i]; j++) - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); mte_wait_after_trig(); err = cur_mte_cxt.fault_valid; /* Check the buffer whether it is filled. */ @@ -82,7 +82,7 @@ static int check_buffer_underflow_by_byte(int mem_type, int mode, /* Set some value in tagged memory and make the buffer underflow */ for (j = sizes[i] - 1; (j >= -underflow_range) && (!cur_mte_cxt.fault_valid); j--) { - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); last_index = j; } mte_wait_after_trig(); @@ -180,7 +180,7 @@ static int check_buffer_overflow_by_byte(int mem_type, int mode, /* Set some value in tagged memory and make the buffer underflow */ for (j = 0, last_index = 0 ; (j < (sizes[i] + overflow_range)) && (cur_mte_cxt.fault_valid == false); j++) { - ptr[j] = '1'; + memset_safe(&ptr[j], '1', 1); last_index = j; } mte_wait_after_trig(); @@ -308,8 +308,8 @@ static int check_buffer_by_block_iterate(int mem_type, int mode, size_t size) result = KSFT_PASS; mte_initialize_current_context(mode, (uintptr_t)dst, size); /* Set some value in memory and copy*/ - memset((void *)src, (int)'1', size); - memcpy((void *)dst, (void *)src, size); + memset_safe((void *)src, (int)'1', size); + memcpy_safe((void *)dst, (void *)src, size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) { result = KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_child_memory.c b/tools/testing/selftests/arm64/mte/check_child_memory.c index e6a8acca2a9411..5c62359e568256 100644 --- a/tools/testing/selftests/arm64/mte/check_child_memory.c +++ b/tools/testing/selftests/arm64/mte/check_child_memory.c @@ -41,7 +41,7 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } else if (child == 0) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); /* Do copy on write */ - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) { fault = 1; @@ -56,14 +56,14 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } } mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memset(ptr - UNDERFLOW, '2', UNDERFLOW); + memset_safe(ptr - UNDERFLOW, '2', UNDERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false) { fault = 1; goto check_child_tag_inheritance_err; } mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memset(ptr + size, '3', OVERFLOW); + memset_safe(ptr + size, '3', OVERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false) { fault = 1; diff --git a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c index 23e4a7a9950c1f..85f1574b88be83 100644 --- a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c +++ b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c @@ -106,7 +106,7 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) } else if (child == 0) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); /* Do copy on write */ - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) { fault = 1; @@ -135,7 +135,7 @@ static int check_child_tag_inheritance(char *ptr, int size, int mode) static int check_mte_memory(char *ptr, int size, int mode, int tag_check) { mte_initialize_current_context(mode, (uintptr_t)ptr, size); - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_mmap_options.c b/tools/testing/selftests/arm64/mte/check_mmap_options.c index 492f2cd41f43b7..0f4d43fe80f2c2 100644 --- a/tools/testing/selftests/arm64/mte/check_mmap_options.c +++ b/tools/testing/selftests/arm64/mte/check_mmap_options.c @@ -72,13 +72,13 @@ static int check_mte_memory(char *ptr, int size, int mode, ptr = mte_insert_atag(ptr); mte_initialize_current_context(mode, (uintptr_t)ptr, size); - memset(ptr, '1', size); + memset_safe(ptr, '1', size); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memset(ptr - UNDERFLOW, '2', UNDERFLOW); + memset_safe(ptr - UNDERFLOW, '2', UNDERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false && tag_check == TAG_CHECK_ON) return KSFT_FAIL; @@ -86,7 +86,7 @@ static int check_mte_memory(char *ptr, int size, int mode, return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memset(ptr + size, '3', OVERFLOW); + memset_safe(ptr + size, '3', OVERFLOW); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == false && tag_check == TAG_CHECK_ON) return KSFT_FAIL; @@ -95,13 +95,13 @@ static int check_mte_memory(char *ptr, int size, int mode, if (tag_op == TAG_OP_STONLY) { mte_initialize_current_context(mode, (uintptr_t)ptr, -UNDERFLOW); - memcpy(buf, ptr - UNDERFLOW, MT_GRANULE_SIZE); + memcpy_safe(buf, ptr - UNDERFLOW, MT_GRANULE_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; mte_initialize_current_context(mode, (uintptr_t)ptr, size + OVERFLOW); - memcpy(buf, ptr + size, MT_GRANULE_SIZE); + memcpy_safe(buf, ptr + size, MT_GRANULE_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid == true) return KSFT_FAIL; diff --git a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c index 6b4fa6705d7c1b..bc39be231ae28e 100644 --- a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c +++ b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c @@ -23,7 +23,7 @@ static int verify_mte_pointer_validity(char *ptr, int mode) { mte_initialize_current_context(mode, (uintptr_t)ptr, BUFFER_SIZE); /* Check the validity of the tagged pointer */ - memset(ptr, '1', BUFFER_SIZE); + memset_safe(ptr, '1', BUFFER_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) { ksft_print_msg("Unexpected fault recorded for %p-%p in mode %x\n", @@ -159,7 +159,7 @@ static int check_none_included_tags(int mem_type, int mode) } mte_initialize_current_context(mode, (uintptr_t)ptr, BUFFER_SIZE); /* Check the write validity of the untagged pointer */ - memset(ptr, '1', BUFFER_SIZE); + memset_safe(ptr, '1', BUFFER_SIZE); mte_wait_after_trig(); if (cur_mte_cxt.fault_valid) break; diff --git a/tools/testing/selftests/arm64/mte/mte_common_util.h b/tools/testing/selftests/arm64/mte/mte_common_util.h index 250d671329a511..547ec5659f78ab 100644 --- a/tools/testing/selftests/arm64/mte/mte_common_util.h +++ b/tools/testing/selftests/arm64/mte/mte_common_util.h @@ -64,6 +64,10 @@ void mte_restore_setup(void); int mte_switch_mode(int mte_option, unsigned long incl_mask, bool stonly); void mte_initialize_current_context(int mode, uintptr_t ptr, ssize_t range); +/* Safe memory access functions */ +void *memset_safe(void *s, int c, size_t n); +void *memcpy_safe(void *dest, const void *src, size_t n); + /* Common utility functions */ int create_temp_file(void); diff --git a/tools/testing/selftests/arm64/mte/mte_helper.S b/tools/testing/selftests/arm64/mte/mte_helper.S index a55dbbc56ed190..eb62abfa1eb4e7 100644 --- a/tools/testing/selftests/arm64/mte/mte_helper.S +++ b/tools/testing/selftests/arm64/mte/mte_helper.S @@ -128,3 +128,47 @@ ENTRY(mte_get_pstate_tco) ubfx x0, x0, #MT_PSTATE_TCO_SHIFT, #1 ret ENDPROC(mte_get_pstate_tco) + +/* + * memset_safe: Fill memory with a constant byte + * Input: + * x0 - destination pointer + * x1 - constant byte + * x2 - number of bytes to fill + * Return: + * x0 - original destination pointer + */ +ENTRY(memset_safe) + cbz x2, 2f + add x4, x2, x0 + mov x3, x0 +1: + strb w1, [x3] + add x3, x3, #0x1 + cmp x3, x4 + b.ne 1b +2: + ret +ENDPROC(memset_safe) + +/* + * memcpy_safe: Copy memory area + * Input: + * x0 - destination pointer + * x1 - source pointer + * x2 - number of bytes to copy + * Return: + * x0 - destination pointer + */ +ENTRY(memcpy_safe) + cbz x2, 2f + mov x3, #0x0 +1: + ldrb w4, [x1, x3] + strb w4, [x0, x3] + add x3, x3, #0x1 + cmp x3, x2 + b.ne 1b +2: + ret +ENDPROC(memcpy_safe) diff --git a/tools/testing/selftests/arm64/signal/Makefile b/tools/testing/selftests/arm64/signal/Makefile index 1381039fb36f98..c4df4be41636c3 100644 --- a/tools/testing/selftests/arm64/signal/Makefile +++ b/tools/testing/selftests/arm64/signal/Makefile @@ -17,8 +17,8 @@ TEST_GEN_PROGS := $(notdir $(PROGS)) # the toplevel makefile include ../../lib.mk -$(TEST_GEN_PROGS): $(PROGS) - cp $(PROGS) $(OUTPUT)/ +$(TEST_GEN_PROGS): $(OUTPUT)/%: testcases/% + cp $< $@ # Common test-unit targets to build common-layout test-cases executables # Needs secondary expansion to properly include the testcase c-file in pre-reqs diff --git a/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c b/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c index e1ccf8f85a70c8..bcef280582ede5 100644 --- a/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c +++ b/tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sve_change_vl.c @@ -2,7 +2,7 @@ /* * Copyright (C) 2021 ARM Limited * - * Attempt to change the SVE vector length in a signal hander, this is not + * Attempt to change the SVE vector length in a signal handler, this is not * supported and is expected to segfault. */ diff --git a/tools/testing/selftests/bpf/prog_tests/rcu_reentry.c b/tools/testing/selftests/bpf/prog_tests/rcu_reentry.c new file mode 100644 index 00000000000000..de23a14b3d408f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/rcu_reentry.c @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Exercise re-entry into call_srcu() from BPF; see progs/rcu_reentry.c. */ +#define _GNU_SOURCE +#include +#include +#include "task_local_storage_helpers.h" +#include "trace_helpers.h" +#include "rcu_reentry.skel.h" + +/* Tiny RCU has no rcu_segcblist_enqueue() to attach to. */ +static bool have_attach_target(void) +{ + unsigned long long addr; + + return kallsyms_find("rcu_segcblist_enqueue", &addr) == 0; +} + +/* Tiny SRCU stubs out srcu_expedite_current(); Tree SRCU exports it. */ +static bool have_tree_srcu(void) +{ + unsigned long long addr; + + return kallsyms_find("srcu_expedite_current", &addr) == 0; +} + +void test_rcu_reentry(void) +{ + struct rcu_reentry *skel; + int err, pidfd = -1, map_fd; + cpu_set_t set, old_set; + bool affinity_saved; + __u64 val = 1; + int cpu; + + if (!have_attach_target()) { + test__skip(); + return; + } + + skel = rcu_reentry__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_and_load")) + return; + + err = rcu_reentry__attach(skel); + if (!ASSERT_OK(err, "skel_attach")) + goto out; + + /* Keep the re-entry on a single CPU; a cpuset may exclude CPU 0. */ + affinity_saved = !sched_getaffinity(0, sizeof(old_set), &old_set); + cpu = sched_getcpu(); + if (!ASSERT_GE(cpu, 0, "getcpu")) + goto out; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (!ASSERT_OK(sched_setaffinity(0, sizeof(set), &set), "setaffinity")) + goto out; + + pidfd = sys_pidfd_open(getpid(), 0); + if (!ASSERT_GE(pidfd, 0, "pidfd_open")) + goto restore; + map_fd = bpf_map__fd(skel->maps.task_stg); + err = bpf_map_update_elem(map_fd, &pidfd, &val, BPF_NOEXIST); + if (!ASSERT_OK(err, "boot_create")) + goto restore; + + /* Arm the handler for this thread, then trigger call_rcu_tasks_trace(). */ + skel->bss->target_pid = syscall(__NR_gettid); + err = bpf_map_delete_elem(map_fd, &pidfd); + if (!ASSERT_OK(err, "boot_delete")) + goto restore; + + /* + * Only Tree SRCU reaches rcu_segcblist_enqueue() from call_srcu(); a + * UP+PREEMPT kernel pairs Tree RCU with Tiny SRCU, so the attach + * succeeds but nothing fires. On Tree SRCU it must fire. + */ + if (!skel->bss->hits) { + if (have_tree_srcu()) + ASSERT_GT(skel->bss->hits, 0, "prog_fired"); + else + test__skip(); + goto restore; + } + ASSERT_EQ(skel->bss->get_errs, 0, "nested_storage_get"); + ASSERT_EQ(skel->bss->del_errs, 0, "nested_storage_delete"); +restore: + if (affinity_saved) + sched_setaffinity(0, sizeof(old_set), &old_set); +out: + if (pidfd >= 0) + close(pidfd); + rcu_reentry__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/prog_tests/test_lsm.c b/tools/testing/selftests/bpf/prog_tests/test_lsm.c index d7495efd4a5621..c0ae813698ae28 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_lsm.c +++ b/tools/testing/selftests/bpf/prog_tests/test_lsm.c @@ -1,18 +1,30 @@ // SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE /* * Copyright (C) 2020 Google LLC. */ #include +#include +#include +#include +#include +#include #include #include +#include + #include "lsm.skel.h" #include "lsm_tailcall.skel.h" char *CMD_ARGS[] = {"true", NULL}; +enum { + INODE_IDMAP_ALL = (1U << 6) - 1, +}; + int exec_cmd(int *monitored_pid) { int child_pid, child_status; @@ -30,6 +42,219 @@ int exec_cmd(int *monitored_pid) return -EINVAL; } +static ssize_t write_nointr(int fd, const void *buf, size_t count) +{ + ssize_t ret; + + do { + ret = write(fd, buf, count); + } while (ret < 0 && errno == EINTR); + + return ret; +} + +static int write_file(const char *path, const char *value) +{ + size_t len = strlen(value); + int fd, saved_errno = 0; + ssize_t ret; + + fd = open(path, O_WRONLY | O_CLOEXEC | O_NOCTTY | O_NOFOLLOW); + if (fd < 0) + return -1; + + ret = write_nointr(fd, value, len); + if (ret < 0) + saved_errno = errno; + else if ((size_t)ret != len) + saved_errno = EIO; + close(fd); + if (saved_errno) { + errno = saved_errno; + return -1; + } + return 0; +} + +static int write_userns_file(pid_t pid, const char *name, const char *value) +{ + char path[64]; + int len; + + len = snprintf(path, sizeof(path), "/proc/%d/%s", pid, name); + if (len < 0 || (size_t)len >= sizeof(path)) { + errno = EOVERFLOW; + return -1; + } + + return write_file(path, value); +} + +static int create_userns_fd(void) +{ + char path[64]; + pid_t pid, waited; + int fd = -1, len, saved_errno, status; + + pid = fork(); + if (pid < 0) + return -1; + if (pid == 0) { + if (unshare(CLONE_NEWUSER)) + _exit(1); + raise(SIGSTOP); + _exit(0); + } + + do { + waited = waitpid(pid, &status, WUNTRACED); + } while (waited < 0 && errno == EINTR); + if (waited != pid) + goto out; + if (!WIFSTOPPED(status)) { + pid = -1; + goto out; + } + + /* A one-entry map is identity for root but remains distinct from nop_mnt_idmap. */ + if (write_userns_file(pid, "setgroups", "deny") && errno != ENOENT) + goto out; + if (write_userns_file(pid, "uid_map", "0 0 1") || + write_userns_file(pid, "gid_map", "0 0 1")) + goto out; + + len = snprintf(path, sizeof(path), "/proc/%d/ns/user", pid); + if (len < 0 || (size_t)len >= sizeof(path)) { + errno = EOVERFLOW; + goto out; + } + fd = open(path, O_RDONLY | O_CLOEXEC); + +out: + saved_errno = errno; + if (pid > 0) { + kill(pid, SIGKILL); + do { + waited = waitpid(pid, NULL, 0); + } while (waited < 0 && errno == EINTR); + } + errno = saved_errno; + return fd; +} + +static int create_idmapped_tmpfs(void) +{ + struct mount_attr attr = { + .attr_set = MOUNT_ATTR_IDMAP, + }; + int fsfd = -1, mntfd = -1, saved_errno, userns_fd = -1; + + userns_fd = create_userns_fd(); + if (userns_fd < 0) + goto out; + + /* A detached tmpfs avoids relying on the host test directory supporting idmaps. */ + fsfd = syscall(__NR_fsopen, "tmpfs", FSOPEN_CLOEXEC); + if (fsfd < 0) + goto out; + if (syscall(__NR_fsconfig, fsfd, FSCONFIG_CMD_CREATE, NULL, NULL, 0)) + goto out; + + mntfd = syscall(__NR_fsmount, fsfd, FSMOUNT_CLOEXEC, 0); + if (mntfd < 0) + goto out; + + attr.userns_fd = userns_fd; + if (syscall(__NR_mount_setattr, mntfd, "", AT_EMPTY_PATH, &attr, + sizeof(attr))) { + close(mntfd); + mntfd = -1; + } + +out: + saved_errno = errno; + if (fsfd >= 0) + close(fsfd); + if (userns_fd >= 0) + close(userns_fd); + errno = saved_errno; + return mntfd; +} + +static int exercise_inode_idmap_hooks(int dirfd) +{ + int fd = -1, ret = -1; + + fd = openat(dirfd, "file", O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, + 0600); + if (!ASSERT_GE(fd, 0, "create")) + goto out; + close(fd); + fd = -1; + + if (!ASSERT_OK(mkdirat(dirfd, "dir", 0700), "mkdir")) + goto out; + if (!ASSERT_OK(symlinkat("target", dirfd, "symlink"), "symlink")) + goto out; + if (!ASSERT_OK(linkat(dirfd, "file", dirfd, "link", 0), "link")) + goto out; + if (!ASSERT_OK(mkfifoat(dirfd, "fifo", 0600), "mknod")) + goto out; + + ret = 0; +out: + if (fd >= 0) + close(fd); + unlinkat(dirfd, "link", 0); + unlinkat(dirfd, "fifo", 0); + unlinkat(dirfd, "symlink", 0); + unlinkat(dirfd, "file", 0); + unlinkat(dirfd, "dir", AT_REMOVEDIR); + return ret; +} + +static int test_lsm_inode_idmap(struct lsm *skel) +{ + char tmpdir[] = "/var/tmp/test_lsm_idmap.XXXXXX"; + __u32 expected = INODE_IDMAP_ALL; + int dirfd = -1, idmapped_dirfd = -1; + int ret = -1; + + if (!ASSERT_OK_PTR(mkdtemp(tmpdir), "mkdtemp")) + return -1; + + dirfd = open(tmpdir, O_RDONLY | O_DIRECTORY | O_CLOEXEC); + if (!ASSERT_GE(dirfd, 0, "open_tmpdir")) + goto out; + + idmapped_dirfd = create_idmapped_tmpfs(); + if (!ASSERT_GE(idmapped_dirfd, 0, "create_idmapped_tmpfs")) + goto out; + + skel->bss->inode_identity_idmap_seen = 0; + skel->bss->inode_idmapped_mount_seen = 0; + + if (!ASSERT_OK(exercise_inode_idmap_hooks(dirfd), "identity_idmap")) + goto out; + if (!ASSERT_OK(exercise_inode_idmap_hooks(idmapped_dirfd), + "idmapped_mount")) + goto out; + + if (!ASSERT_EQ(skel->bss->inode_identity_idmap_seen, expected, + "inode_identity_idmap_seen")) + goto out; + ret = ASSERT_EQ(skel->bss->inode_idmapped_mount_seen, expected, + "inode_idmapped_mount_seen") ? 0 : -1; + +out: + if (idmapped_dirfd >= 0) + close(idmapped_dirfd); + if (dirfd >= 0) + close(dirfd); + rmdir(tmpdir); + return ret; +} + static int test_lsm(struct lsm *skel) { struct bpf_link *link; @@ -53,6 +278,10 @@ static int test_lsm(struct lsm *skel) skel->bss->monitored_pid = getpid(); + err = test_lsm_inode_idmap(skel); + if (!ASSERT_OK(err, "test_lsm_inode_idmap")) + return err; + err = stack_mprotect(); if (!ASSERT_EQ(err, -1, "stack_mprotect") || !ASSERT_EQ(errno, EPERM, "stack_mprotect")) @@ -71,6 +300,8 @@ static int test_lsm(struct lsm *skel) skel->bss->copy_test = 0; skel->bss->bprm_count = 0; skel->bss->mprotect_count = 0; + skel->bss->inode_identity_idmap_seen = 0; + skel->bss->inode_idmapped_mount_seen = 0; return 0; } diff --git a/tools/testing/selftests/bpf/progs/lsm.c b/tools/testing/selftests/bpf/progs/lsm.c index 7de173daf27b6d..7e32fbddbfadf0 100644 --- a/tools/testing/selftests/bpf/progs/lsm.c +++ b/tools/testing/selftests/bpf/progs/lsm.c @@ -84,6 +84,85 @@ char _license[] SEC("license") = "GPL"; int monitored_pid = 0; int mprotect_count = 0; int bprm_count = 0; +__u32 inode_identity_idmap_seen = 0; +__u32 inode_idmapped_mount_seen = 0; + +enum { + INODE_IDMAP_CREATE = 1U << 0, + INODE_IDMAP_LINK = 1U << 1, + INODE_IDMAP_SYMLINK = 1U << 2, + INODE_IDMAP_MKDIR = 1U << 3, + INODE_IDMAP_MKNOD = 1U << 4, + INODE_IDMAP_PERMISSION = 1U << 5, +}; + +static __always_inline bool is_monitored_idmap(struct mnt_idmap *idmap) +{ + __u32 pid = bpf_get_current_pid_tgid() >> 32; + + return monitored_pid == pid && idmap; +} + +static __always_inline bool is_identity_idmap(struct mnt_idmap *idmap) +{ + return idmap->uid_map.nr_extents == 0 && + idmap->gid_map.nr_extents == 0; +} + +static __always_inline int record_inode_idmap(struct mnt_idmap *idmap, + __u32 hook, int ret) +{ + if (ret || !is_monitored_idmap(idmap)) + return ret; + if (is_identity_idmap(idmap)) + inode_identity_idmap_seen |= hook; + else + inode_idmapped_mount_seen |= hook; + return 0; +} + +SEC("lsm/inode_create") +int BPF_PROG(test_inode_create, struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_CREATE, ret); +} + +SEC("lsm/inode_link") +int BPF_PROG(test_inode_link, struct mnt_idmap *idmap, + struct dentry *old_dentry, struct inode *dir, + struct dentry *new_dentry, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_LINK, ret); +} + +SEC("lsm/inode_symlink") +int BPF_PROG(test_inode_symlink, struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, const char *old_name, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_SYMLINK, ret); +} + +SEC("lsm/inode_mkdir") +int BPF_PROG(test_inode_mkdir, struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_MKDIR, ret); +} + +SEC("lsm/inode_mknod") +int BPF_PROG(test_inode_mknod, struct mnt_idmap *idmap, struct inode *dir, + struct dentry *dentry, umode_t mode, dev_t dev, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_MKNOD, ret); +} + +SEC("lsm/inode_permission") +int BPF_PROG(test_inode_permission, struct mnt_idmap *idmap, + struct inode *inode, int mask, int ret) +{ + return record_inode_idmap(idmap, INODE_IDMAP_PERMISSION, ret); +} SEC("lsm/file_mprotect") int BPF_PROG(test_int_hook, struct vm_area_struct *vma, diff --git a/tools/testing/selftests/bpf/progs/rcu_reentry.c b/tools/testing/selftests/bpf/progs/rcu_reentry.c new file mode 100644 index 00000000000000..47a36f704cf3e1 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rcu_reentry.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Re-enter call_srcu() from a BPF program. fentry on rcu_segcblist_enqueue() + * fires inside call_srcu()'s enqueue (reached from srcu_gp_start_if_needed() + * with the srcu_data ->lock held); the handler then calls call_rcu_tasks_trace() + * -- itself call_srcu() on rcu_tasks_trace_srcu_struct -- re-entering the same + * srcu_data on the same CPU. + */ +#include "vmlinux.h" +#include +#include + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, __u64); +} task_stg SEC(".maps"); + +int target_pid; +int hits; +int get_errs; +int del_errs; +int done; + +SEC("fentry/rcu_segcblist_enqueue") +int BPF_PROG(reenter) +{ + struct task_struct *cur; + + if (done || !target_pid) + return 0; + + cur = bpf_get_current_task_btf(); + if (cur->pid != target_pid) + return 0; + + /* Issue the nested call exactly once, so the test is deterministic. */ + done = 1; + __sync_fetch_and_add(&hits, 1); + + /* Re-enter via a task-storage delete, which calls call_rcu_tasks_trace(). */ + if (!bpf_task_storage_get(&task_stg, cur, 0, BPF_LOCAL_STORAGE_GET_F_CREATE)) + __sync_fetch_and_add(&get_errs, 1); + else if (bpf_task_storage_delete(&task_stg, cur)) + __sync_fetch_and_add(&del_errs, 1); + + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/test_siphash.h b/tools/testing/selftests/bpf/progs/test_siphash.h index 5d3a7ec3678082..9a85670a9deae7 100644 --- a/tools/testing/selftests/bpf/progs/test_siphash.h +++ b/tools/testing/selftests/bpf/progs/test_siphash.h @@ -22,7 +22,7 @@ static inline u64 rol64(u64 word, unsigned int shift) #define SIPHASH_CONST_2 0x6c7967656e657261ULL #define SIPHASH_CONST_3 0x7465646279746573ULL -/* lib/siphash.c */ +/* lib/crypto/siphash.c */ #define SIPROUND SIPHASH_PERMUTATION(v0, v1, v2, v3) #define PREAMBLE(len) \ diff --git a/tools/testing/selftests/clone3/clone3_clear_sighand.c b/tools/testing/selftests/clone3/clone3_clear_sighand.c index de0c9d62015de8..0ad62b80ada208 100644 --- a/tools/testing/selftests/clone3/clone3_clear_sighand.c +++ b/tools/testing/selftests/clone3/clone3_clear_sighand.c @@ -50,12 +50,12 @@ static void test_clone3_clear_sighand(void) * Check that CLONE_CLEAR_SIGHAND and CLONE_SIGHAND are mutually * exclusive. */ - args.flags |= CLONE_CLEAR_SIGHAND | CLONE_SIGHAND; + args.flags |= CLONE_VM | CLONE_CLEAR_SIGHAND | CLONE_SIGHAND; args.exit_signal = SIGCHLD; pid = sys_clone3(&args, sizeof(args)); - if (pid > 0) + if (pid != -1 || errno != EINVAL) ksft_exit_fail_msg( - "clone3(CLONE_CLEAR_SIGHAND | CLONE_SIGHAND) succeeded\n"); + "clone3(CLONE_CLEAR_SIGHAND | CLONE_SIGHAND) did not fail with EINVAL\n"); act.sa_handler = nop_handler; ret = sigemptyset(&act.sa_mask); diff --git a/tools/testing/selftests/core/close_range_test.c b/tools/testing/selftests/core/close_range_test.c index f14eca63f20c40..20ecb65e529b01 100644 --- a/tools/testing/selftests/core/close_range_test.c +++ b/tools/testing/selftests/core/close_range_test.c @@ -36,6 +36,14 @@ static inline int sys_close_range(unsigned int fd, unsigned int max_fd, return syscall(__NR_close_range, fd, max_fd, flags); } +static void clear_cloexec(const int *fds, size_t n) +{ + size_t i; + + for (i = 0; i < n; i++) + fcntl(fds[i], F_SETFD, 0); +} + TEST(core_close_range) { int i, ret; @@ -236,6 +244,50 @@ TEST(close_range_unshare_capped) EXPECT_EQ(0, WEXITSTATUS(status)); } +TEST(close_range_unshare_hole) +{ + int i, status; + pid_t pid; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + /* Fill the first two words of the table. */ + for (i = 3; i < 128; i++) + ASSERT_GE(dup2(0, i), 0); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + /* Punch a hole into the second word, behind a full first one. */ + if (sys_close_range(70, 80, CLOSE_RANGE_UNSHARE)) + exit(EXIT_FAILURE); + + for (i = 3; i < 128; i++) { + bool closed = i >= 70 && i <= 80; + + if (closed == (fcntl(i, F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + /* A stale full bit on word 1 would hand out 128, not 70. */ + if (dup(0) != 70) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The shared table the child unshared from is untouched. */ + for (i = 3; i < 128; i++) + EXPECT_NE(-1, fcntl(i, F_GETFD)); +} + TEST(close_range_cloexec) { int i, ret; @@ -593,6 +645,905 @@ TEST(close_range_cloexec_unshare_syzbot) EXPECT_EQ(close(fd3), 0); } +TEST(close_range_except) +{ + int i, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_EXCEPT"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); + + /* The bounds are checked before the range is turned around. */ + EXPECT_EQ(-1, sys_close_range(open_fds[20], open_fds[10], + CLOSE_RANGE_EXCEPT)); + EXPECT_EQ(EINVAL, errno); + + /* Everything above open_fds[50] goes. */ + ASSERT_EQ(0, sys_close_range(0, open_fds[50], CLOSE_RANGE_EXCEPT)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(i <= 50, fcntl(open_fds[i], F_GETFD) != -1); + + /* A window in the middle takes stdio with it, so do that in a fork. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i <= 50; i++) { + bool kept = i >= 10 && i <= 20; + + if (kept != (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + if (fcntl(STDERR_FILENO, F_GETFD) != -1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The fork had a table of its own. */ + for (i = 0; i <= 50; i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_except_cloexec) +{ + int i, ret; + int open_fds[101]; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_CLOEXEC | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_EXCEPT"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool inside = i >= 10 && i <= 20; + int flags = fcntl(open_fds[i], F_GETFD); + + EXPECT_NE(-1, flags); + EXPECT_EQ(inside ? 0 : FD_CLOEXEC, flags & FD_CLOEXEC); + } + + /* stdio sits outside of the window too. */ + EXPECT_EQ(FD_CLOEXEC, fcntl(STDERR_FILENO, F_GETFD) & FD_CLOEXEC); + + /* A window that starts at 0 marks only what lies above it. */ + clear_cloexec(open_fds, ARRAY_SIZE(open_fds)); + ASSERT_EQ(0, fcntl(STDERR_FILENO, F_SETFD, 0)); + ASSERT_EQ(0, sys_close_range(0, open_fds[20], + CLOSE_RANGE_CLOEXEC | CLOSE_RANGE_EXCEPT)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(i <= 20 ? 0 : FD_CLOEXEC, + fcntl(open_fds[i], F_GETFD) & FD_CLOEXEC); + EXPECT_EQ(0, fcntl(STDERR_FILENO, F_GETFD) & FD_CLOEXEC); + + /* One at the top marks only what lies below it, stdio included. */ + clear_cloexec(open_fds, ARRAY_SIZE(open_fds)); + ASSERT_EQ(0, sys_close_range(open_fds[80], UINT_MAX, + CLOSE_RANGE_CLOEXEC | CLOSE_RANGE_EXCEPT)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(i < 80 ? FD_CLOEXEC : 0, + fcntl(open_fds[i], F_GETFD) & FD_CLOEXEC); + EXPECT_EQ(FD_CLOEXEC, fcntl(STDERR_FILENO, F_GETFD) & FD_CLOEXEC); + + /* One that cannot hold a descriptor marks everything. */ + clear_cloexec(open_fds, ARRAY_SIZE(open_fds)); + ASSERT_EQ(0, fcntl(STDERR_FILENO, F_SETFD, 0)); + ASSERT_EQ(0, sys_close_range(UINT_MAX, UINT_MAX, + CLOSE_RANGE_CLOEXEC | CLOSE_RANGE_EXCEPT)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(FD_CLOEXEC, fcntl(open_fds[i], F_GETFD) & FD_CLOEXEC); + EXPECT_EQ(FD_CLOEXEC, fcntl(STDERR_FILENO, F_GETFD) & FD_CLOEXEC); +} + +TEST(close_range_except_cloexec_unshare) +{ + int i, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything marks nothing. */ + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_EXCEPT"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(0, fcntl(open_fds[i], F_GETFD) & FD_CLOEXEC); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_UNSHARE | CLOSE_RANGE_CLOEXEC | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool inside = i >= 10 && i <= 20; + int flags = fcntl(open_fds[i], F_GETFD); + + if (flags == -1) + exit(EXIT_FAILURE); + if ((flags & FD_CLOEXEC) != (inside ? 0 : FD_CLOEXEC)) + exit(EXIT_FAILURE); + } + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The shared table the child unshared from is untouched. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(0, fcntl(open_fds[i], F_GETFD) & FD_CLOEXEC); +} + +TEST(close_range_except_bounds) +{ + int i, c, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_EXCEPT"); + } + ASSERT_EQ(0, ret); + + struct { + unsigned int fd, max_fd, flags; + } cases[] = { + /* A window at the top drops everything below it. */ + { open_fds[50], UINT_MAX, CLOSE_RANGE_EXCEPT }, + /* One that cannot hold a descriptor keeps nothing. */ + { UINT_MAX, UINT_MAX, CLOSE_RANGE_EXCEPT }, + /* One of a single descriptor keeps just that. */ + { open_fds[30], open_fds[30], CLOSE_RANGE_EXCEPT }, + /* The unshare form on a table that is not shared acts in place. */ + { open_fds[10], open_fds[20], + CLOSE_RANGE_UNSHARE | CLOSE_RANGE_EXCEPT }, + }; + + /* Each of them takes stdio with it, so do that in a fork. */ + for (c = 0; c < ARRAY_SIZE(cases); c++) { + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(cases[c].fd, cases[c].max_fd, + cases[c].flags); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + unsigned int fd = open_fds[i]; + bool kept = fd >= cases[c].fd && + fd <= cases[c].max_fd; + + if (kept != (fcntl(fd, F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + if (fcntl(STDERR_FILENO, F_GETFD) != -1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + } + + /* Each fork had a table of its own. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_except_unshare) +{ + int i, ret, status; + pid_t pid; + int open_fds[200]; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + /* Odd slots are close-on-exec, which makes no difference here. */ + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_EXCEPT"); + } + ASSERT_EQ(0, ret); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + /* The window sits near the top, so the clone is sized off its end. */ + ret = sys_close_range(open_fds[150], open_fds[160], + CLOSE_RANGE_UNSHARE | CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool kept = i >= 150 && i <= 160; + + if (kept != (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + if (fcntl(STDERR_FILENO, F_GETFD) != -1) + exit(EXIT_FAILURE); + + /* What was left behind is handed out again, from the bottom. */ + if (dup(open_fds[150]) != 0) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* A window at the bottom keeps just that, stdio included. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(0, open_fds[10], + CLOSE_RANGE_UNSHARE | CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + if ((i <= 10) != (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + if (fcntl(STDERR_FILENO, F_GETFD) == -1) + exit(EXIT_FAILURE); + + /* The first slot left behind is the next one handed out. */ + if (dup(0) != open_fds[10] + 1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* A window that cannot hold a descriptor keeps nothing. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(UINT_MAX, UINT_MAX, + CLOSE_RANGE_UNSHARE | CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + if (fcntl(open_fds[i], F_GETFD) != -1) + exit(EXIT_FAILURE); + + if (fcntl(STDERR_FILENO, F_GETFD) != -1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The shared table the child unshared from is untouched. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_cloexec_only) +{ + int i, ret; + int open_fds[101]; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + /* Odd slots are close-on-exec, even ones are not. */ + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_CLOEXEC_ONLY); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool closed = i % 2 && i >= 10 && i <= 20; + + EXPECT_EQ(!closed, fcntl(open_fds[i], F_GETFD) != -1); + } + + /* A range above the table closes nothing. */ + ASSERT_EQ(0, sys_close_range(UINT_MAX, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool closed = i % 2 && i >= 10 && i <= 20; + + EXPECT_EQ(!closed, fcntl(open_fds[i], F_GETFD) != -1); + } + + /* Do what an exec would do to the rest. */ + ASSERT_EQ(0, sys_close_range(0, UINT_MAX, CLOSE_RANGE_CLOEXEC_ONLY)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(!(i % 2), fcntl(open_fds[i], F_GETFD) != -1); +} + +TEST(close_range_cloexec_only_except) +{ + int i, ret; + int open_fds[101]; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool kept = !(i % 2) || (i >= 10 && i <= 20); + int flags = i % 2 ? FD_CLOEXEC : 0; + + /* The kept ones keep their flag, so exec still drops them. */ + EXPECT_EQ(kept ? flags : -1, fcntl(open_fds[i], F_GETFD)); + } + + /* A range that cannot hold an open descriptor keeps nothing. */ + ASSERT_EQ(0, sys_close_range(UINT_MAX, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT)); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_EQ(!(i % 2), fcntl(open_fds[i], F_GETFD) != -1); +} + +TEST(close_range_cloexec_only_except_bounds) +{ + int i, c, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + struct { + unsigned int fd, max_fd; + } cases[] = { + /* A window at the top keeps the marked ones in it. */ + { open_fds[80], UINT_MAX }, + /* One at the bottom keeps the marked ones in it. */ + { 0, open_fds[20] }, + /* One that cannot hold a descriptor keeps none of them. */ + { UINT_MAX, UINT_MAX }, + }; + + for (c = 0; c < ARRAY_SIZE(cases); c++) { + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(cases[c].fd, cases[c].max_fd, + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + unsigned int fd = open_fds[i]; + bool kept = !(i % 2) || (fd >= cases[c].fd && + fd <= cases[c].max_fd); + int flags = i % 2 ? FD_CLOEXEC : 0; + + if (fcntl(fd, F_GETFD) != (kept ? flags : -1)) + exit(EXIT_FAILURE); + } + + /* stdio is neither marked nor gone. */ + if (fcntl(STDERR_FILENO, F_GETFD) & FD_CLOEXEC) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + } + + /* Each fork had a table of its own. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_cloexec_only_unshare) +{ + int i, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + ASSERT_NE(-1, fcntl(open_fds[i], F_GETFD)); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool closed = i % 2 && i >= 10 && i <= 20; + + if (closed == (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* A range at the top keeps the descriptors without the flag in it. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[50], UINT_MAX, + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool closed = i % 2 && i >= 50; + + if (closed == (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + /* The first slot left behind is the next one handed out. */ + if (dup(0) != open_fds[51]) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The shared table the child unshared from is untouched. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_cloexec_only_except_unshare) +{ + int i, ret, status; + pid_t pid; + int open_fds[101]; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY | (i % 2 ? O_CLOEXEC : 0)); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + /* A range covering everything keeps everything. */ + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + ASSERT_NE(-1, fcntl(open_fds[i], F_GETFD)); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[10], open_fds[20], + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool kept = !(i % 2) || (i >= 10 && i <= 20); + int flags = i % 2 ? FD_CLOEXEC : 0; + + if (fcntl(open_fds[i], F_GETFD) != (kept ? flags : -1)) + exit(EXIT_FAILURE); + } + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* A window that cannot hold a descriptor keeps none of the marked. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(UINT_MAX, UINT_MAX, + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + if ((i % 2) == (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + + if (fcntl(STDERR_FILENO, F_GETFD) == -1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* One that covers everything keeps everything, in a clone too. */ + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + if (fcntl(open_fds[i], F_GETFD) != (i % 2 ? FD_CLOEXEC : 0)) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); + + /* The shared table the child unshared from is untouched. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) + EXPECT_NE(-1, fcntl(open_fds[i], F_GETFD)); +} + +TEST(close_range_cloexec_only_except_unshare_sizing) +{ + int i, ret, status; + pid_t pid; + int open_fds[200]; + struct __clone_args args = { + .flags = CLONE_FILES, + .exit_signal = SIGCHLD, + }; + + /* All close-on-exec, so the kept range alone sizes the clone. */ + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + int fd; + + fd = open("/dev/null", O_RDONLY | O_CLOEXEC); + ASSERT_GE(fd, 0) { + if (errno == ENOENT) + SKIP(return, "Skipping test since /dev/null does not exist"); + } + + open_fds[i] = fd; + } + + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + pid = sys_clone3(&args, sizeof(args)); + ASSERT_GE(pid, 0); + + if (pid == 0) { + ret = sys_close_range(open_fds[150], open_fds[160], + CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT); + if (ret) + exit(EXIT_FAILURE); + + for (i = 0; i < ARRAY_SIZE(open_fds); i++) { + bool kept = i >= 150 && i <= 160; + + if (kept != (fcntl(open_fds[i], F_GETFD) != -1)) + exit(EXIT_FAILURE); + } + + /* Nothing set close-on-exec on stdio. */ + if (fcntl(STDERR_FILENO, F_GETFD) == -1) + exit(EXIT_FAILURE); + + exit(EXIT_SUCCESS); + } + + EXPECT_EQ(waitpid(pid, &status, 0), pid); + EXPECT_EQ(true, WIFEXITED(status)); + EXPECT_EQ(0, WEXITSTATUS(status)); +} + +TEST(close_range_cloexec_only_einval) +{ + int ret; + + /* A range covering everything keeps everything, so this only probes. */ + ret = sys_close_range(0, UINT_MAX, + CLOSE_RANGE_CLOEXEC_ONLY | CLOSE_RANGE_EXCEPT); + if (ret < 0) { + if (errno == ENOSYS) + SKIP(return, "close_range() syscall not supported"); + if (errno == EINVAL) + SKIP(return, "close_range() doesn't support CLOSE_RANGE_CLOEXEC_ONLY"); + } + ASSERT_EQ(0, ret); + + EXPECT_EQ(-1, sys_close_range(3, UINT_MAX, CLOSE_RANGE_CLOEXEC | + CLOSE_RANGE_CLOEXEC_ONLY)); + EXPECT_EQ(EINVAL, errno); + + /* The other flags do not make the pair acceptable. */ + EXPECT_EQ(-1, sys_close_range(3, UINT_MAX, CLOSE_RANGE_UNSHARE | + CLOSE_RANGE_CLOEXEC | + CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT)); + EXPECT_EQ(EINVAL, errno); + + /* The bounds are checked with the new flag too. */ + EXPECT_EQ(-1, sys_close_range(4, 3, CLOSE_RANGE_CLOEXEC_ONLY | + CLOSE_RANGE_EXCEPT)); + EXPECT_EQ(EINVAL, errno); +} + TEST(close_range_bitmap_corruption) { pid_t pid; diff --git a/tools/testing/selftests/coredump/.gitignore b/tools/testing/selftests/coredump/.gitignore index 097f52db0be964..e32f6e9006f6e2 100644 --- a/tools/testing/selftests/coredump/.gitignore +++ b/tools/testing/selftests/coredump/.gitignore @@ -2,3 +2,5 @@ stackdump_test coredump_socket_test coredump_socket_protocol_test +coredump_signal_test +coredump_worker_test diff --git a/tools/testing/selftests/coredump/Makefile b/tools/testing/selftests/coredump/Makefile index dece1a31d561b0..c4cdab7b7d0ba8 100644 --- a/tools/testing/selftests/coredump/Makefile +++ b/tools/testing/selftests/coredump/Makefile @@ -3,7 +3,12 @@ CFLAGS += -Wall -O0 -g $(KHDR_INCLUDES) $(TOOLS_INCLUDES) TEST_GEN_PROGS := stackdump_test \ coredump_socket_test \ - coredump_socket_protocol_test + coredump_socket_protocol_test \ + coredump_notify_signal_test \ + coredump_signal_test \ + coredump_worker_test +# Spawned by the kernel as the |helper, not a test of its own. +TEST_GEN_FILES := coredump_notify_signal_helper TEST_FILES := stackdump include ../lib.mk @@ -11,3 +16,7 @@ include ../lib.mk $(OUTPUT)/stackdump_test: coredump_test_helpers.c $(OUTPUT)/coredump_socket_test: coredump_test_helpers.c $(OUTPUT)/coredump_socket_protocol_test: coredump_test_helpers.c +$(OUTPUT)/coredump_notify_signal_test: coredump_test_helpers.c +$(OUTPUT)/coredump_notify_signal_helper: coredump_test_helpers.c +$(OUTPUT)/coredump_signal_test: coredump_test_helpers.c +$(OUTPUT)/coredump_worker_test: coredump_test_helpers.c diff --git a/tools/testing/selftests/coredump/coredump_notify_signal.h b/tools/testing/selftests/coredump/coredump_notify_signal.h new file mode 100644 index 00000000000000..92868a43425cbc --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_notify_signal.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __COREDUMP_NOTIFY_SIGNAL_H +#define __COREDUMP_NOTIFY_SIGNAL_H + +#include +#include + +/* + * Define a bunch of constants we need. We create a situation where the + * NT_FILE note blows past 200K. That's way beyond the default 64K + * pipe ring and past the ~36K an af_unix skb holds. So we force a write + * to come back short. + */ +#define NOTIFY_SIGNAL_MAP_COUNT 4000 +#define NOTIFY_SIGNAL_ANON_BYTES (4UL << 20) +#define NOTIFY_SIGNAL_STALL_US 200000 +#define NOTIFY_SIGNAL_MAPFILE "/tmp/coredump.notify_signal.mapfile" +#define NOTIFY_SIGNAL_TRIGGER "/tmp/coredump.notify_signal.trigger" +#define NOTIFY_SIGNAL_CORE_FILE "/tmp/coredump.notify_signal.core" +#define NOTIFY_SIGNAL_CORE_TMPFILE "/tmp/coredump.notify_signal.core.tmp" +#define NOTIFY_SIGNAL_SOCKET "/tmp/coredump.notify_signal.socket" + +void crashing_child_notify_signal(void); +bool coredump_io_uring_available(void); +ssize_t recv_coredump_notify_signal(int fd, int fd_out, bool arm); +long long coredump_expected_size(const char *path); + +#endif /* __COREDUMP_NOTIFY_SIGNAL_H */ diff --git a/tools/testing/selftests/coredump/coredump_notify_signal_helper.c b/tools/testing/selftests/coredump/coredump_notify_signal_helper.c new file mode 100644 index 00000000000000..849f5c1ea7365b --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_notify_signal_helper.c @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * The |helper half of coredump_notify_signal_test. The kernel spawns this + * with the coredump on stdin, so it cannot be part of the test binary. + * It saves the dump and, once the notes have started, trips the fifo the + * crashing task is polling so TIF_NOTIFY_SIGNAL is raised while the note + * write is in flight. + */ + +#include +#include +#include +#include + +#include "coredump_notify_signal.h" + +int main(int argc, char *argv[]) +{ + int fd_core_file; + ssize_t ret; + + fd_core_file = open(NOTIFY_SIGNAL_CORE_TMPFILE, + O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0600); + if (fd_core_file < 0) { + fprintf(stderr, "%s: open failed: %m\n", argv[0]); + return EXIT_FAILURE; + } + + ret = recv_coredump_notify_signal(STDIN_FILENO, fd_core_file, true); + close(fd_core_file); + if (ret < 0) + goto err; + + /* The test polls for this name, so only create it once it is whole. */ + if (rename(NOTIFY_SIGNAL_CORE_TMPFILE, NOTIFY_SIGNAL_CORE_FILE)) { + fprintf(stderr, "%s: rename failed: %m\n", argv[0]); + goto err; + } + + return EXIT_SUCCESS; + +err: + unlink(NOTIFY_SIGNAL_CORE_TMPFILE); + return EXIT_FAILURE; +} diff --git a/tools/testing/selftests/coredump/coredump_notify_signal_test.c b/tools/testing/selftests/coredump/coredump_notify_signal_test.c new file mode 100644 index 00000000000000..4a98ab141c419f --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_notify_signal_test.c @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * A coredump is meant to be interrupted by SIGKILL and by the freezer and + * by nothing else. dump_interrupted() says so, but the blocking waits + * underneath it test signal_pending(), which is also true for + * TIF_NOTIFY_SIGNAL. A crashing task that has an io_uring completion land + * on it mid-dump therefore keeps dumping while every wait it enters bails + * out at once, and the dump is silently cut short. Nothing reports it: + * binfmt_elf sets has_dumped before it writes anything, so WCOREDUMP() + * says the dump worked. + * + * A coredump note is the only dump_emit() that exceeds what the transport + * takes in one go, so it is the one write that is certain to block. Both + * tests crash a child holding enough file backed mappings for its NT_FILE + * note to run past that, arm an io_uring poll on it, and trip the poll + * while the note is being written. What comes out has to be the whole + * dump. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "coredump_test.h" + +FIXTURE_SETUP(coredump) +{ + FILE *file; + int ret; + + self->pid_coredump_server = -ESRCH; + self->fd_tmpfs_detached = -1; + file = fopen("/proc/sys/kernel/core_pattern", "r"); + ASSERT_NE(NULL, file); + + ret = fread(self->original_core_pattern, 1, + sizeof(self->original_core_pattern), file); + ASSERT_TRUE(ret || feof(file)); + ASSERT_LT(ret, sizeof(self->original_core_pattern)); + + self->original_core_pattern[ret] = '\0'; + self->fd_tmpfs_detached = create_detached_tmpfs(); + ASSERT_GE(self->fd_tmpfs_detached, 0); + + ret = fclose(file); + ASSERT_EQ(0, ret); + + /* A stale core file from a killed previous run would fake a pass. */ + unlink(NOTIFY_SIGNAL_CORE_FILE); + /* And a stale socket would fail the server's bind. */ + unlink(NOTIFY_SIGNAL_SOCKET); + unlink(NOTIFY_SIGNAL_TRIGGER); + ASSERT_EQ(mkfifo(NOTIFY_SIGNAL_TRIGGER, 0600), 0); +} + +FIXTURE_TEARDOWN(coredump) +{ + const char *reason; + FILE *file; + int ret, status; + + if (self->pid_coredump_server > 0) { + kill(self->pid_coredump_server, SIGTERM); + waitpid(self->pid_coredump_server, &status, 0); + } + unlink(NOTIFY_SIGNAL_CORE_FILE); + unlink(NOTIFY_SIGNAL_CORE_TMPFILE); + unlink(NOTIFY_SIGNAL_SOCKET); + unlink(NOTIFY_SIGNAL_TRIGGER); + unlink(NOTIFY_SIGNAL_MAPFILE); + + file = fopen("/proc/sys/kernel/core_pattern", "w"); + if (!file) { + reason = "Unable to open core_pattern"; + goto fail; + } + + ret = fprintf(file, "%s", self->original_core_pattern); + if (ret < 0) { + reason = "Unable to write to core_pattern"; + goto fail; + } + + ret = fclose(file); + if (ret) { + reason = "Unable to close core_pattern"; + goto fail; + } + + if (self->fd_tmpfs_detached >= 0) { + ret = close(self->fd_tmpfs_detached); + if (ret < 0) { + reason = "Unable to close detached tmpfs"; + goto fail; + } + self->fd_tmpfs_detached = -1; + } + + return; +fail: + /* This should never happen */ + fprintf(stderr, "Failed to cleanup coredump test: %s\n", reason); +} + +/* Check that what the helper or the server saved is the whole dump. */ +static void check_whole_coredump(struct __test_metadata *const _metadata) +{ + long long expected; + struct stat st; + + expected = coredump_expected_size(NOTIFY_SIGNAL_CORE_FILE); + ASSERT_GT(expected, 0); + ASSERT_EQ(stat(NOTIFY_SIGNAL_CORE_FILE, &st), 0); + ASSERT_EQ((long long)st.st_size, expected); +} + +/* + * The dump goes to a |helper, so the reader is a separate program the + * kernel spawns. It saves what it received to NOTIFY_SIGNAL_CORE_FILE. + */ +TEST_F(coredump, notify_signal_pipe) +{ + char pattern[PATH_MAX], helper[PATH_MAX], *p; + struct stat st; + int status, i; + pid_t pid; + ssize_t n; + + if (!coredump_io_uring_available()) + SKIP(return, "io_uring not available"); + + n = readlink("/proc/self/exe", helper, sizeof(helper) - 1); + ASSERT_GT(n, 0); + helper[n] = '\0'; + p = strstr(helper, "coredump_notify_signal_test"); + ASSERT_NE(p, NULL); + ASSERT_LE((size_t)(p - helper) + sizeof("coredump_notify_signal_helper"), + sizeof(helper)); + strcpy(p, "coredump_notify_signal_helper"); + if (access(helper, X_OK)) + SKIP(return, "coredump_notify_signal_helper not built"); + + ASSERT_LT(snprintf(pattern, sizeof(pattern), "|%s", helper), + (int)sizeof(pattern)); + ASSERT_TRUE(set_core_pattern(pattern)); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_notify_signal(); + + ASSERT_EQ(waitpid(pid, &status, 0), pid); + ASSERT_TRUE(WIFSIGNALED(status)); + + /* The kernel does not wait for the helper, so poll for it. */ + for (i = 0; i < 100; i++) { + if (!stat(NOTIFY_SIGNAL_CORE_FILE, &st) && st.st_size) + break; + usleep(100000); + } + + check_whole_coredump(_metadata); +} + +/* The same thing with the dump going to a coredump socket. */ +TEST_F(coredump, notify_signal_socket) +{ + pid_t pid, pid_coredump_server; + int ipc_sockets[2], status; + char pattern[PATH_MAX]; + char c; + + if (!coredump_io_uring_available()) + SKIP(return, "io_uring not available"); + + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, + ipc_sockets), 0); + ASSERT_LT(snprintf(pattern, sizeof(pattern), "@%s", + NOTIFY_SIGNAL_SOCKET), (int)sizeof(pattern)); + ASSERT_TRUE(set_core_pattern(pattern)); + + pid_coredump_server = fork(); + ASSERT_GE(pid_coredump_server, 0); + if (pid_coredump_server == 0) { + int fd_server = -1, fd_coredump = -1, fd_core_file = -1; + int exit_code = EXIT_FAILURE; + + close(ipc_sockets[0]); + + fd_server = create_and_listen_unix_socket(NOTIFY_SIGNAL_SOCKET); + if (fd_server < 0) + goto out; + if (write_nointr(ipc_sockets[1], "1", 1) < 0) + goto out; + close(ipc_sockets[1]); + + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) + goto out; + + fd_core_file = open(NOTIFY_SIGNAL_CORE_FILE, + O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, + 0600); + if (fd_core_file < 0) + goto out; + + if (recv_coredump_notify_signal(fd_coredump, fd_core_file, + true) < 0) + goto out; + + exit_code = EXIT_SUCCESS; +out: + if (fd_core_file >= 0) + close(fd_core_file); + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_server >= 0) + close(fd_server); + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); + ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); + EXPECT_EQ(close(ipc_sockets[0]), 0); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_notify_signal(); + + ASSERT_EQ(waitpid(pid, &status, 0), pid); + ASSERT_TRUE(WIFSIGNALED(status)); + + wait_and_check_coredump_server(pid_coredump_server, _metadata, self); + + check_whole_coredump(_metadata); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/coredump/coredump_signal_test.c b/tools/testing/selftests/coredump/coredump_signal_test.c new file mode 100644 index 00000000000000..fdf48b1447811f --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_signal_test.c @@ -0,0 +1,238 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "coredump_test.h" + +/* Big enough to fill the socket buffer many times over. */ +#define CRASH_MAPPING_SIZE (32 * 1024 * 1024) + +FIXTURE_SETUP(coredump) +{ + FILE *file; + int ret; + + self->pid_coredump_server = -ESRCH; + self->fd_tmpfs_detached = -1; + file = fopen("/proc/sys/kernel/core_pattern", "r"); + ASSERT_NE(NULL, file); + + ret = fread(self->original_core_pattern, 1, sizeof(self->original_core_pattern), file); + ASSERT_TRUE(ret || feof(file)); + ASSERT_LT(ret, sizeof(self->original_core_pattern)); + + self->original_core_pattern[ret] = '\0'; + + ret = fclose(file); + ASSERT_EQ(0, ret); +} + +FIXTURE_TEARDOWN(coredump) +{ + const char *reason; + FILE *file; + int ret, status; + + if (self->pid_coredump_server > 0) { + kill(self->pid_coredump_server, SIGTERM); + waitpid(self->pid_coredump_server, &status, 0); + } + unlink("/tmp/coredump.file"); + unlink("/tmp/coredump.socket"); + + file = fopen("/proc/sys/kernel/core_pattern", "w"); + if (!file) { + reason = "Unable to open core_pattern"; + goto fail; + } + + ret = fprintf(file, "%s", self->original_core_pattern); + if (ret < 0) { + reason = "Unable to write to core_pattern"; + goto fail; + } + + ret = fclose(file); + if (ret) { + reason = "Unable to close core_pattern"; + goto fail; + } + + return; +fail: + /* This should never happen */ + fprintf(stderr, "Failed to cleanup coredump test: %s\n", reason); +} + +static volatile int waiter_ready; + +static void usr1_handler(int sig) +{ +} + +/* Sleeps in sigtimedwait() with SIGUSR1 unblocked only inside the kernel. */ +static void *sigwaiter(void *arg) +{ + sigset_t set; + siginfo_t info; + + sigemptyset(&set); + sigaddset(&set, SIGUSR1); + __atomic_store_n(&waiter_ready, 1, __ATOMIC_RELEASE); + for (;;) + sigtimedwait(&set, &info, NULL); + return NULL; +} + +/* + * Crash with a shared SIGUSR1 still queued for this thread. The vfork() + * child queues it while we sleep killably and then the SIGSEGV that is + * dequeued first. The zap wakes the sibling out of sigtimedwait() and its + * mask restore retargets SIGUSR1 to the dumper. + */ +static void crashing_child_retarget(bool queue_shared) +{ + sigset_t all, old; + pthread_t thread; + pid_t pid, tid; + char *p; + + p = mmap(NULL, CRASH_MAPPING_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + _exit(EXIT_FAILURE); + memset(p, 0x5a, CRASH_MAPPING_SIZE); + + signal(SIGUSR1, usr1_handler); + sigfillset(&all); + pthread_sigmask(SIG_BLOCK, &all, &old); + /* One waiter only, retarget stops at the first thread not blocking it. */ + if (pthread_create(&thread, NULL, sigwaiter, NULL)) + _exit(EXIT_FAILURE); + pthread_sigmask(SIG_SETMASK, &old, NULL); + while (!__atomic_load_n(&waiter_ready, __ATOMIC_ACQUIRE)) + usleep(1000); + usleep(50 * 1000); + + pid = getpid(); + tid = syscall(SYS_gettid); + if (vfork() == 0) { + if (queue_shared) + syscall(SYS_kill, pid, SIGUSR1); + syscall(SYS_tgkill, pid, tid, SIGSEGV); + syscall(SYS_exit, 0); + } + + /* Not reached, the pending SIGSEGV dumps core. */ + for (;;) + pause(); +} + +/* + * Dump the crashing child into /tmp/coredump.file through a server that + * holds the read back so the dumper blocks on the full socket buffer. + */ +static void run_coredump(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, bool queue_shared) +{ + pid_t pid, pid_coredump_server; + int ipc_sockets[2]; + int status; + char c; + + unlink("/tmp/coredump.file"); + unlink("/tmp/coredump.socket"); + ASSERT_TRUE(set_core_pattern("@/tmp/coredump.socket")); + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + + pid_coredump_server = fork(); + ASSERT_GE(pid_coredump_server, 0); + if (pid_coredump_server == 0) { + int fd_server = -1, fd_coredump = -1, fd_core_file = -1; + int exit_code = EXIT_FAILURE; + + close(ipc_sockets[0]); + + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); + if (fd_server < 0) + goto out; + + if (write_nointr(ipc_sockets[1], "1", 1) < 0) + goto out; + close(ipc_sockets[1]); + + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) { + fprintf(stderr, "%s: accept4 failed: %m\n", __func__); + goto out; + } + + /* Let the dumper run into the full socket buffer first. */ + sleep(1); + + fd_core_file = creat("/tmp/coredump.file", 0644); + if (fd_core_file < 0) { + fprintf(stderr, "%s: creat failed: %m\n", __func__); + goto out; + } + + if (recv_coredump_bytes(fd_coredump, fd_core_file) < 0) + goto out; + + exit_code = EXIT_SUCCESS; +out: + if (fd_core_file >= 0) + close(fd_core_file); + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_server >= 0) + close(fd_server); + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); + ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); + EXPECT_EQ(close(ipc_sockets[0]), 0); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_retarget(queue_shared); + + waitpid(pid, &status, 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_EQ(WTERMSIG(status), SIGSEGV); + ASSERT_TRUE(WCOREDUMP(status)); + + wait_and_check_coredump_server(pid_coredump_server, _metadata, self); +} + +static void check_coredump_complete(struct __test_metadata *const _metadata) +{ + int fd; + + fd = open("/tmp/coredump.file", O_RDONLY | O_CLOEXEC); + ASSERT_GE(fd, 0); + ASSERT_TRUE(check_coredump_extent(fd)); + close(fd); +} + +TEST_F(coredump, retarget_shared_pending) +{ + run_coredump(_metadata, self, false); + check_coredump_complete(_metadata); + + run_coredump(_metadata, self, true); + check_coredump_complete(_metadata); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/coredump/coredump_socket_protocol_test.c b/tools/testing/selftests/coredump/coredump_socket_protocol_test.c index d9fa6239b5a9fd..f5c9bad8754634 100644 --- a/tools/testing/selftests/coredump/coredump_socket_protocol_test.c +++ b/tools/testing/selftests/coredump/coredump_socket_protocol_test.c @@ -151,9 +151,7 @@ TEST_F(coredump, socket_request_kernel) goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { + if (!check_coredump_req(&req)) { fprintf(stderr, "socket_request_kernel: check_coredump_req failed\n"); goto out; } @@ -301,9 +299,7 @@ TEST_F(coredump, socket_request_userspace) goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { + if (!check_coredump_req(&req)) { fprintf(stderr, "socket_request_userspace: check_coredump_req failed\n"); goto out; } @@ -441,9 +437,7 @@ TEST_F(coredump, socket_request_reject) goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { + if (!check_coredump_req(&req)) { fprintf(stderr, "socket_request_reject: check_coredump_req failed\n"); goto out; } @@ -514,93 +508,89 @@ TEST_F(coredump, socket_request_reject) wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } -TEST_F(coredump, socket_request_invalid_flag_combination) +/* An ack the kernel must refuse and how. */ +struct refused_ack { + /* The ack, and how many bytes of it the server sends before it hangs up. */ + struct coredump_ack ack; + size_t bytes; + /* The marker the kernel answers with, or none if @no_marker. */ + enum coredump_mark mark; + bool no_marker; +}; + +/* Send @refused, expect the kernel to refuse it and hang up. */ +static void check_refused_ack(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, + const struct refused_ack *refused) { - int pidfd, ret, status; + int pidfd, status; pid_t pid, pid_coredump_server; struct pidfd_info info = {}; int ipc_sockets[2]; char c; + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); - ret = socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets); - ASSERT_EQ(ret, 0); - pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - struct coredump_req req = {}; int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; close(ipc_sockets[0]); fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); - if (fd_server < 0) { - fprintf(stderr, "socket_request_invalid_flag_combination: create_and_listen_unix_socket failed: %m\n"); + if (fd_server < 0) goto out; - } - if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_request_invalid_flag_combination: write_nointr to ipc socket failed: %m\n"); + if (write_nointr(ipc_sockets[1], "1", 1) < 0) goto out; - } close(ipc_sockets[1]); fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - fprintf(stderr, "socket_request_invalid_flag_combination: accept4 failed: %m\n"); + if (fd_coredump < 0) goto out; - } fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_request_invalid_flag_combination: get_peer_pidfd failed\n"); + if (fd_peer_pidfd < 0) goto out; - } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_request_invalid_flag_combination: get_pidfd_info failed\n"); + /* The task shows as dumping while it waits for the ack. */ + if (!get_pidfd_info(fd_peer_pidfd, &info)) goto out; - } - if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_request_invalid_flag_combination: PIDFD_INFO_COREDUMP not set in mask\n"); + if (!(info.mask & PIDFD_INFO_COREDUMP) || + !(info.coredump_mask & PIDFD_COREDUMPED)) { + fprintf(stderr, "Peer isn't marked as dumping\n"); goto out; } - if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_request_invalid_flag_combination: PIDFD_COREDUMPED not set in coredump_mask\n"); + if (!read_coredump_req(fd_coredump, &req)) goto out; - } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_request_invalid_flag_combination: read_coredump_req failed\n"); + if (!check_coredump_req(&req)) goto out; - } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "socket_request_invalid_flag_combination: check_coredump_req failed\n"); + if (!send_coredump_ack_bytes(fd_coredump, &refused->ack, + refused->bytes)) goto out; - } - if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_KERNEL | COREDUMP_REJECT | COREDUMP_WAIT, 0)) { - fprintf(stderr, "socket_request_invalid_flag_combination: send_coredump_ack failed\n"); + /* Nothing more to say. A server that died looks the same. */ + if (shutdown(fd_coredump, SHUT_WR)) goto out; - } - if (!read_marker(fd_coredump, COREDUMP_MARK_CONFLICTING)) { - fprintf(stderr, "socket_request_invalid_flag_combination: read_marker COREDUMP_MARK_CONFLICTING failed\n"); + if (!refused->no_marker && + !read_marker(fd_coredump, refused->mark)) + goto out; + + /* The kernel hangs up after a refusal, marker or not. */ + if (!read_hangup(fd_coredump)) goto out; - } exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_request_invalid_flag_combination: completed successfully\n"); out: if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); @@ -635,7 +625,83 @@ TEST_F(coredump, socket_request_invalid_flag_combination) wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } +/* Ack @ack_mask, expect the kernel to refuse it as conflicting. */ +static void check_conflicting_ack(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, __u64 ack_mask) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .mask = ack_mask, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_CONFLICTING, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* More than one of KERNEL, USERSPACE and REJECT. */ +TEST_F(coredump, socket_request_invalid_flag_combination) +{ + check_conflicting_ack(_metadata, self, + COREDUMP_KERNEL | COREDUMP_REJECT | COREDUMP_WAIT); +} + +/* A flag the kernel didn't advertise in coredump_req->mask. */ TEST_F(coredump, socket_request_unknown_flag) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .mask = 1ULL << 63, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_UNSUPPORTED, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* An ack smaller than the first published struct. */ +TEST_F(coredump, socket_request_invalid_size_small) +{ + struct refused_ack refused = { + .ack = { + .size = COREDUMP_ACK_SIZE_VER0 / 2, + .mask = COREDUMP_REJECT | COREDUMP_WAIT, + }, + .bytes = COREDUMP_ACK_SIZE_VER0 / 2, + .mark = COREDUMP_MARK_MINSIZE, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* An ack bigger than the kernel said it accepts. */ +TEST_F(coredump, socket_request_invalid_size_large) +{ + struct refused_ack refused = { + .ack = { + .size = COREDUMP_ACK_SIZE_VER0 + PAGE_SIZE, + .mask = COREDUMP_REJECT | COREDUMP_WAIT, + }, + .bytes = COREDUMP_ACK_SIZE_VER0 + PAGE_SIZE, + .mark = COREDUMP_MARK_MAXSIZE, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* + * Test: PIDFD_INFO_COREDUMP_SIGNAL via socket coredump with SIGSEGV + * + * Verify that when using socket-based coredump protocol, + * the coredump_signal field is correctly exposed as SIGSEGV. + * Also check that the coredump_code field is correctly exposed + * as SEGV_MAPERR. + */ +TEST_F(coredump, socket_coredump_signal_sigsegv) { int pidfd, ret, status; pid_t pid, pid_coredump_server; @@ -659,12 +725,12 @@ TEST_F(coredump, socket_request_unknown_flag) fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); if (fd_server < 0) { - fprintf(stderr, "socket_request_unknown_flag: create_and_listen_unix_socket failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: create_and_listen_unix_socket failed: %m\n"); goto out; } if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_request_unknown_flag: write_nointr to ipc socket failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: write_nointr to ipc socket failed: %m\n"); goto out; } @@ -672,55 +738,73 @@ TEST_F(coredump, socket_request_unknown_flag) fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); if (fd_coredump < 0) { - fprintf(stderr, "socket_request_unknown_flag: accept4 failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: accept4 failed: %m\n"); goto out; } fd_peer_pidfd = get_peer_pidfd(fd_coredump); if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_request_unknown_flag: get_peer_pidfd failed\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: get_peer_pidfd failed\n"); goto out; } if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_request_unknown_flag: get_pidfd_info failed\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: get_pidfd_info failed\n"); goto out; } if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_request_unknown_flag: PIDFD_INFO_COREDUMP not set in mask\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP not set in mask\n"); goto out; } if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_request_unknown_flag: PIDFD_COREDUMPED not set in coredump_mask\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_COREDUMPED not set in coredump_mask\n"); goto out; } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_request_unknown_flag: read_coredump_req failed\n"); + /* Verify coredump_signal is available and correct */ + if (!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)) { + fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP_SIGNAL not set in mask\n"); + goto out; + } + + if (info.coredump_signal != SIGSEGV) { + fprintf(stderr, "socket_coredump_signal_sigsegv: coredump_signal=%d, expected SIGSEGV=%d\n", + info.coredump_signal, SIGSEGV); + goto out; + } + + /* Verify coredump_code is available and correct */ + if (!(info.mask & PIDFD_INFO_COREDUMP_CODE)) { + fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP_CODE not set in mask\n"); + goto out; + } + + if (info.coredump_code != SEGV_MAPERR) { + fprintf(stderr, "socket_coredump_signal_sigsegv: coredump_code=%d, expected SEGV_MAPERR=%d\n", + info.coredump_code, SEGV_MAPERR); goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "socket_request_unknown_flag: check_coredump_req failed\n"); + if (!read_coredump_req(fd_coredump, &req)) { + fprintf(stderr, "socket_coredump_signal_sigsegv: read_coredump_req failed\n"); goto out; } - if (!send_coredump_ack(fd_coredump, &req, (1ULL << 63), 0)) { - fprintf(stderr, "socket_request_unknown_flag: send_coredump_ack failed\n"); + if (!send_coredump_ack(fd_coredump, &req, + COREDUMP_REJECT | COREDUMP_WAIT, 0)) { + fprintf(stderr, "socket_coredump_signal_sigsegv: send_coredump_ack failed\n"); goto out; } - if (!read_marker(fd_coredump, COREDUMP_MARK_UNSUPPORTED)) { - fprintf(stderr, "socket_request_unknown_flag: read_marker COREDUMP_MARK_UNSUPPORTED failed\n"); + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { + fprintf(stderr, "socket_coredump_signal_sigsegv: read_marker COREDUMP_MARK_REQACK failed\n"); goto out; } exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_request_unknown_flag: completed successfully\n"); + fprintf(stderr, "socket_coredump_signal_sigsegv: completed successfully\n"); out: if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); @@ -746,16 +830,27 @@ TEST_F(coredump, socket_request_unknown_flag) waitpid(pid, &status, 0); ASSERT_TRUE(WIFSIGNALED(status)); - ASSERT_FALSE(WCOREDUMP(status)); + ASSERT_EQ(WTERMSIG(status), SIGSEGV); ASSERT_TRUE(get_pidfd_info(pidfd, &info)); - ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); - ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP)); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)); + ASSERT_EQ(info.coredump_signal, SIGSEGV); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_CODE)); + ASSERT_EQ(info.coredump_code, SEGV_MAPERR); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } -TEST_F(coredump, socket_request_invalid_size_small) +/* + * Test: PIDFD_INFO_COREDUMP_SIGNAL via socket coredump with SIGABRT + * + * Verify that when using socket-based coredump protocol, + * the coredump_signal field is correctly exposed as SIGABRT. + * Also check that the coredump_code field is correctly exposed + * as SI_TKILL. + */ +TEST_F(coredump, socket_coredump_signal_sigabrt) { int pidfd, ret, status; pid_t pid, pid_coredump_server; @@ -779,12 +874,12 @@ TEST_F(coredump, socket_request_invalid_size_small) fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); if (fd_server < 0) { - fprintf(stderr, "socket_request_invalid_size_small: create_and_listen_unix_socket failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: create_and_listen_unix_socket failed: %m\n"); goto out; } if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_request_invalid_size_small: write_nointr to ipc socket failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: write_nointr to ipc socket failed: %m\n"); goto out; } @@ -792,57 +887,67 @@ TEST_F(coredump, socket_request_invalid_size_small) fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); if (fd_coredump < 0) { - fprintf(stderr, "socket_request_invalid_size_small: accept4 failed: %m\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: accept4 failed: %m\n"); goto out; } fd_peer_pidfd = get_peer_pidfd(fd_coredump); if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_request_invalid_size_small: get_peer_pidfd failed\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: get_peer_pidfd failed\n"); goto out; } if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_request_invalid_size_small: get_pidfd_info failed\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: get_pidfd_info failed\n"); goto out; } if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_request_invalid_size_small: PIDFD_INFO_COREDUMP not set in mask\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_INFO_COREDUMP not set in mask\n"); goto out; } if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_request_invalid_size_small: PIDFD_COREDUMPED not set in coredump_mask\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_COREDUMPED not set in coredump_mask\n"); goto out; } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_request_invalid_size_small: read_coredump_req failed\n"); + /* Verify coredump_signal is available and correct */ + if (!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)) { + fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_INFO_COREDUMP_SIGNAL not set in mask\n"); + goto out; + } + + if (info.coredump_signal != SIGABRT) { + fprintf(stderr, "socket_coredump_signal_sigabrt: coredump_signal=%d, expected SIGABRT=%d\n", + info.coredump_signal, SIGABRT); + goto out; + } + + if (info.coredump_code != SI_TKILL) { + fprintf(stderr, "socket_coredump_signal_sigabrt: coredump_code=%d, expected SI_TKILL=%d\n", + info.coredump_code, SI_TKILL); goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "socket_request_invalid_size_small: check_coredump_req failed\n"); + if (!read_coredump_req(fd_coredump, &req)) { + fprintf(stderr, "socket_coredump_signal_sigabrt: read_coredump_req failed\n"); goto out; } if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_REJECT | COREDUMP_WAIT, - COREDUMP_ACK_SIZE_VER0 / 2)) { - fprintf(stderr, "socket_request_invalid_size_small: send_coredump_ack failed\n"); + COREDUMP_REJECT | COREDUMP_WAIT, 0)) { + fprintf(stderr, "socket_coredump_signal_sigabrt: send_coredump_ack failed\n"); goto out; } - if (!read_marker(fd_coredump, COREDUMP_MARK_MINSIZE)) { - fprintf(stderr, "socket_request_invalid_size_small: read_marker COREDUMP_MARK_MINSIZE failed\n"); + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { + fprintf(stderr, "socket_coredump_signal_sigabrt: read_marker COREDUMP_MARK_REQACK failed\n"); goto out; } exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_request_invalid_size_small: completed successfully\n"); + fprintf(stderr, "socket_coredump_signal_sigabrt: completed successfully\n"); out: if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); @@ -861,257 +966,388 @@ TEST_F(coredump, socket_request_invalid_size_small) pid = fork(); ASSERT_GE(pid, 0); if (pid == 0) - crashing_child(); + abort(); pidfd = sys_pidfd_open(pid, 0); ASSERT_GE(pidfd, 0); waitpid(pid, &status, 0); ASSERT_TRUE(WIFSIGNALED(status)); - ASSERT_FALSE(WCOREDUMP(status)); + ASSERT_EQ(WTERMSIG(status), SIGABRT); ASSERT_TRUE(get_pidfd_info(pidfd, &info)); - ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); - ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP)); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)); + ASSERT_EQ(info.coredump_signal, SIGABRT); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_CODE)); + ASSERT_EQ(info.coredump_code, SI_TKILL); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } -TEST_F(coredump, socket_request_invalid_size_large) +TEST_F_TIMEOUT(coredump, socket_multiple_crashing_coredumps, 500) { - int pidfd, ret, status; - pid_t pid, pid_coredump_server; + int pidfd[NUM_CRASHING_COREDUMPS], status[NUM_CRASHING_COREDUMPS]; + pid_t pid[NUM_CRASHING_COREDUMPS], pid_coredump_server; struct pidfd_info info = {}; int ipc_sockets[2]; char c; ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); - ret = socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets); - ASSERT_EQ(ret, 0); + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - struct coredump_req req = {}; - int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1, fd_core_file = -1; int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; close(ipc_sockets[0]); - fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); if (fd_server < 0) { - fprintf(stderr, "socket_request_invalid_size_large: create_and_listen_unix_socket failed: %m\n"); + fprintf(stderr, "Failed to create and listen on unix socket\n"); goto out; } if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_request_invalid_size_large: write_nointr to ipc socket failed: %m\n"); + fprintf(stderr, "Failed to notify parent via ipc socket\n"); goto out; } - close(ipc_sockets[1]); - fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - fprintf(stderr, "socket_request_invalid_size_large: accept4 failed: %m\n"); - goto out; - } + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) { + fprintf(stderr, "accept4 failed: %m\n"); + goto out; + } - fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_request_invalid_size_large: get_peer_pidfd failed\n"); - goto out; - } + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) { + fprintf(stderr, "get_peer_pidfd failed for fd %d: %m\n", fd_coredump); + goto out; + } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_request_invalid_size_large: get_pidfd_info failed\n"); - goto out; - } + if (!get_pidfd_info(fd_peer_pidfd, &info)) { + fprintf(stderr, "get_pidfd_info failed for fd %d\n", fd_peer_pidfd); + goto out; + } - if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_request_invalid_size_large: PIDFD_INFO_COREDUMP not set in mask\n"); - goto out; - } + if (!(info.mask & PIDFD_INFO_COREDUMP)) { + fprintf(stderr, "pidfd info missing PIDFD_INFO_COREDUMP for fd %d\n", fd_peer_pidfd); + goto out; + } + if (!(info.coredump_mask & PIDFD_COREDUMPED)) { + fprintf(stderr, "pidfd info missing PIDFD_COREDUMPED for fd %d\n", fd_peer_pidfd); + goto out; + } - if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_request_invalid_size_large: PIDFD_COREDUMPED not set in coredump_mask\n"); - goto out; - } + if (!read_coredump_req(fd_coredump, &req)) { + fprintf(stderr, "read_coredump_req failed for fd %d\n", fd_coredump); + goto out; + } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_request_invalid_size_large: read_coredump_req failed\n"); - goto out; - } + if (!check_coredump_req(&req)) { + fprintf(stderr, "check_coredump_req failed for fd %d\n", fd_coredump); + goto out; + } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "socket_request_invalid_size_large: check_coredump_req failed\n"); - goto out; - } + if (!send_coredump_ack(fd_coredump, &req, + COREDUMP_KERNEL | COREDUMP_WAIT, 0)) { + fprintf(stderr, "send_coredump_ack failed for fd %d\n", fd_coredump); + goto out; + } - if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_REJECT | COREDUMP_WAIT, - COREDUMP_ACK_SIZE_VER0 + PAGE_SIZE)) { - fprintf(stderr, "socket_request_invalid_size_large: send_coredump_ack failed\n"); - goto out; - } + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { + fprintf(stderr, "read_marker failed for fd %d\n", fd_coredump); + goto out; + } - if (!read_marker(fd_coredump, COREDUMP_MARK_MAXSIZE)) { - fprintf(stderr, "socket_request_invalid_size_large: read_marker COREDUMP_MARK_MAXSIZE failed\n"); - goto out; - } + fd_core_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_core_file < 0) { + fprintf(stderr, "%m - open_coredump_tmpfile failed for fd %d\n", fd_coredump); + goto out; + } - exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_request_invalid_size_large: completed successfully\n"); -out: - if (fd_peer_pidfd >= 0) - close(fd_peer_pidfd); - if (fd_coredump >= 0) - close(fd_coredump); - if (fd_server >= 0) - close(fd_server); - _exit(exit_code); - } - self->pid_coredump_server = pid_coredump_server; + for (;;) { + char buffer[4096]; + ssize_t bytes_read, bytes_write; - EXPECT_EQ(close(ipc_sockets[1]), 0); + bytes_read = read(fd_coredump, buffer, sizeof(buffer)); + if (bytes_read < 0) { + fprintf(stderr, "read failed for fd %d: %m\n", fd_coredump); + goto out; + } + + if (bytes_read == 0) + break; + + bytes_write = write(fd_core_file, buffer, bytes_read); + if (bytes_read != bytes_write) { + if (bytes_write < 0 && errno == ENOSPC) + continue; + fprintf(stderr, "write failed for fd %d: %m\n", fd_core_file); + goto out; + } + } + + close(fd_core_file); + close(fd_peer_pidfd); + close(fd_coredump); + fd_peer_pidfd = -1; + fd_coredump = -1; + } + + exit_code = EXIT_SUCCESS; +out: + if (fd_core_file >= 0) + close(fd_core_file); + if (fd_peer_pidfd >= 0) + close(fd_peer_pidfd); + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_server >= 0) + close(fd_server); + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); EXPECT_EQ(close(ipc_sockets[0]), 0); - pid = fork(); - ASSERT_GE(pid, 0); - if (pid == 0) - crashing_child(); + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + pid[i] = fork(); + ASSERT_GE(pid[i], 0); + if (pid[i] == 0) + crashing_child(); + pidfd[i] = sys_pidfd_open(pid[i], 0); + ASSERT_GE(pidfd[i], 0); + } - pidfd = sys_pidfd_open(pid, 0); - ASSERT_GE(pidfd, 0); + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + waitpid(pid[i], &status[i], 0); + ASSERT_TRUE(WIFSIGNALED(status[i])); + ASSERT_TRUE(WCOREDUMP(status[i])); + } - waitpid(pid, &status, 0); - ASSERT_TRUE(WIFSIGNALED(status)); - ASSERT_FALSE(WCOREDUMP(status)); + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + info.mask = PIDFD_INFO_EXIT | PIDFD_INFO_COREDUMP; + ASSERT_EQ(ioctl(pidfd[i], PIDFD_GET_INFO, &info), 0); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + } - ASSERT_TRUE(get_pidfd_info(pidfd, &info)); - ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); - ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + wait_and_check_coredump_server(pid_coredump_server, _metadata, self); +} + +TEST_F_TIMEOUT(coredump, socket_multiple_crashing_coredumps_epoll_workers, 500) +{ + int pidfd[NUM_CRASHING_COREDUMPS], status[NUM_CRASHING_COREDUMPS]; + pid_t pid[NUM_CRASHING_COREDUMPS], pid_coredump_server, worker_pids[NUM_CRASHING_COREDUMPS]; + struct pidfd_info info = {}; + int ipc_sockets[2]; + char c; + + ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + + pid_coredump_server = fork(); + ASSERT_GE(pid_coredump_server, 0); + if (pid_coredump_server == 0) { + int fd_server = -1, exit_code = EXIT_FAILURE, n_conns = 0; + fd_server = -1; + exit_code = EXIT_FAILURE; + n_conns = 0; + close(ipc_sockets[0]); + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); + if (fd_server < 0) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: create_and_listen_unix_socket failed: %m\n"); + goto out; + } + + if (write_nointr(ipc_sockets[1], "1", 1) < 0) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: write_nointr to ipc socket failed: %m\n"); + goto out; + } + close(ipc_sockets[1]); + + while (n_conns < NUM_CRASHING_COREDUMPS) { + int fd_coredump = -1, fd_peer_pidfd = -1, fd_core_file = -1; + struct coredump_req req = {}; + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) + continue; + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: accept4 failed: %m\n"); + goto out; + } + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: get_peer_pidfd failed\n"); + goto out; + } + if (!get_pidfd_info(fd_peer_pidfd, &info)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: get_pidfd_info failed\n"); + goto out; + } + if (!(info.mask & PIDFD_INFO_COREDUMP) || !(info.coredump_mask & PIDFD_COREDUMPED)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: missing PIDFD_INFO_COREDUMP or PIDFD_COREDUMPED\n"); + goto out; + } + if (!read_coredump_req(fd_coredump, &req)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: read_coredump_req failed\n"); + goto out; + } + if (!check_coredump_req(&req)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: check_coredump_req failed\n"); + goto out; + } + if (!send_coredump_ack(fd_coredump, &req, COREDUMP_KERNEL | COREDUMP_WAIT, 0)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: send_coredump_ack failed\n"); + goto out; + } + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: read_marker failed\n"); + goto out; + } + fd_core_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_core_file < 0) { + fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: open_coredump_tmpfile failed: %m\n"); + goto out; + } + pid_t worker = fork(); + if (worker == 0) { + close(fd_server); + process_coredump_worker(fd_coredump, fd_peer_pidfd, fd_core_file); + } + worker_pids[n_conns] = worker; + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_peer_pidfd >= 0) + close(fd_peer_pidfd); + if (fd_core_file >= 0) + close(fd_core_file); + n_conns++; + } + exit_code = EXIT_SUCCESS; +out: + if (fd_server >= 0) + close(fd_server); + + // Reap all worker processes + for (int i = 0; i < n_conns; i++) { + int wstatus; + if (waitpid(worker_pids[i], &wstatus, 0) < 0) { + fprintf(stderr, "Failed to wait for worker %d: %m\n", worker_pids[i]); + } else if (WIFEXITED(wstatus) && WEXITSTATUS(wstatus) != EXIT_SUCCESS) { + fprintf(stderr, "Worker %d exited with error code %d\n", worker_pids[i], WEXITSTATUS(wstatus)); + exit_code = EXIT_FAILURE; + } + } + + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); + ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); + EXPECT_EQ(close(ipc_sockets[0]), 0); + + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + pid[i] = fork(); + ASSERT_GE(pid[i], 0); + if (pid[i] == 0) + crashing_child(); + pidfd[i] = sys_pidfd_open(pid[i], 0); + ASSERT_GE(pidfd[i], 0); + } + + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + ASSERT_GE(waitpid(pid[i], &status[i], 0), 0); + ASSERT_TRUE(WIFSIGNALED(status[i])); + ASSERT_TRUE(WCOREDUMP(status[i])); + } + + for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { + info.mask = PIDFD_INFO_EXIT | PIDFD_INFO_COREDUMP; + ASSERT_EQ(ioctl(pidfd[i], PIDFD_GET_INFO, &info), 0); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + } wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } /* - * Test: PIDFD_INFO_COREDUMP_SIGNAL via socket coredump with SIGSEGV - * - * Verify that when using socket-based coredump protocol, - * the coredump_signal field is correctly exposed as SIGSEGV. - * Also check that the coredump_code field is correctly exposed - * as SEGV_MAPERR. + * Reassemble a record stream and check that what comes out is an ELF + * core file. The records themselves are validated by recv_coredump_records(). */ -TEST_F(coredump, socket_coredump_signal_sigsegv) +TEST_F(coredump, socket_request_sparse_reassemble) { - int pidfd, ret, status; + int fd_core_file, pidfd, status; pid_t pid, pid_coredump_server; struct pidfd_info info = {}; int ipc_sockets[2]; char c; + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); - ret = socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets); - ASSERT_EQ(ret, 0); - pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - struct coredump_req req = {}; int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_file = -1; int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; close(ipc_sockets[0]); fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); - if (fd_server < 0) { - fprintf(stderr, "socket_coredump_signal_sigsegv: create_and_listen_unix_socket failed: %m\n"); + if (fd_server < 0) goto out; - } - if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_coredump_signal_sigsegv: write_nointr to ipc socket failed: %m\n"); + if (write_nointr(ipc_sockets[1], "1", 1) < 0) goto out; - } close(ipc_sockets[1]); fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - fprintf(stderr, "socket_coredump_signal_sigsegv: accept4 failed: %m\n"); + if (fd_coredump < 0) goto out; - } fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_coredump_signal_sigsegv: get_peer_pidfd failed\n"); + if (fd_peer_pidfd < 0) goto out; - } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: get_pidfd_info failed\n"); + fd_file = creat("/tmp/coredump.file", 0644); + if (fd_file < 0) goto out; - } - if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP not set in mask\n"); + if (!read_coredump_req(fd_coredump, &req)) goto out; - } - if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_COREDUMPED not set in coredump_mask\n"); + if (!check_coredump_req(&req)) goto out; - } - /* Verify coredump_signal is available and correct */ - if (!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP_SIGNAL not set in mask\n"); + if (!send_coredump_ack(fd_coredump, &req, + COREDUMP_KERNEL | COREDUMP_RECORDS | + COREDUMP_SPARSE | COREDUMP_WAIT, 0)) goto out; - } - if (info.coredump_signal != SIGSEGV) { - fprintf(stderr, "socket_coredump_signal_sigsegv: coredump_signal=%d, expected SIGSEGV=%d\n", - info.coredump_signal, SIGSEGV); + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) goto out; - } - /* Verify coredump_code is available and correct */ - if (!(info.mask & PIDFD_INFO_COREDUMP_CODE)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: PIDFD_INFO_COREDUMP_CODE not set in mask\n"); + if (recv_coredump_records(fd_coredump, fd_file, NULL, NULL, -1) < 0) goto out; - } - - if (info.coredump_code != SEGV_MAPERR) { - fprintf(stderr, "socket_coredump_signal_sigsegv: coredump_code=%d, expected SEGV_MAPERR=%d\n", - info.coredump_code, SEGV_MAPERR); - goto out; - } - - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: read_coredump_req failed\n"); - goto out; - } - - if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_REJECT | COREDUMP_WAIT, 0)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: send_coredump_ack failed\n"); - goto out; - } - - if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { - fprintf(stderr, "socket_coredump_signal_sigsegv: read_marker COREDUMP_MARK_REQACK failed\n"); - goto out; - } exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_coredump_signal_sigsegv: completed successfully\n"); out: + if (fd_file >= 0) + close(fd_file); if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); if (fd_coredump >= 0) @@ -1136,125 +1372,113 @@ TEST_F(coredump, socket_coredump_signal_sigsegv) waitpid(pid, &status, 0); ASSERT_TRUE(WIFSIGNALED(status)); - ASSERT_EQ(WTERMSIG(status), SIGSEGV); + ASSERT_TRUE(WCOREDUMP(status)); ASSERT_TRUE(get_pidfd_info(pidfd, &info)); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP)); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)); - ASSERT_EQ(info.coredump_signal, SIGSEGV); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_CODE)); - ASSERT_EQ(info.coredump_code, SEGV_MAPERR); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); + + /* What the records reassemble into has to be an ELF core file. */ + fd_core_file = open("/tmp/coredump.file", O_RDONLY | O_CLOEXEC); + ASSERT_GE(fd_core_file, 0); + ASSERT_TRUE(is_elf_core(fd_core_file)); + EXPECT_EQ(close(fd_core_file), 0); } /* - * Test: PIDFD_INFO_COREDUMP_SIGNAL via socket coredump with SIGABRT - * - * Verify that when using socket-based coredump protocol, - * the coredump_signal field is correctly exposed as SIGABRT. - * Also check that the coredump_code field is correctly exposed - * as SI_TKILL. + * Crash a child with a mostly-unpopulated mapping and reassemble its + * record stream, reporting what crossed the socket and the coredump + * size the records describe. With @kill_peer the server kills the task + * once the coredump is under way so the kernel has to cut it short. */ -TEST_F(coredump, socket_coredump_signal_sigabrt) +static void check_record_dump(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, __u64 ack_mask, + bool kill_peer, ssize_t *received, + off_t *coredump_size) { - int pidfd, ret, status; + bool truncated = false; + int pidfd, status; pid_t pid, pid_coredump_server; struct pidfd_info info = {}; int ipc_sockets[2]; + int pipefds[2]; char c; + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + ASSERT_EQ(pipe(pipefds), 0); ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); - ret = socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets); - ASSERT_EQ(ret, 0); - pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - struct coredump_req req = {}; int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_file = -1; int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; + bool is_truncated = false; + off_t size = 0; + ssize_t ret; close(ipc_sockets[0]); + close(pipefds[0]); fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); - if (fd_server < 0) { - fprintf(stderr, "socket_coredump_signal_sigabrt: create_and_listen_unix_socket failed: %m\n"); + if (fd_server < 0) goto out; - } - if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_coredump_signal_sigabrt: write_nointr to ipc socket failed: %m\n"); + if (write_nointr(ipc_sockets[1], "1", 1) < 0) goto out; - } close(ipc_sockets[1]); fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - fprintf(stderr, "socket_coredump_signal_sigabrt: accept4 failed: %m\n"); + if (fd_coredump < 0) goto out; - } fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_coredump_signal_sigabrt: get_peer_pidfd failed\n"); + if (fd_peer_pidfd < 0) goto out; - } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: get_pidfd_info failed\n"); + /* + * The reassembled coredump is bigger than the mapping the + * child made, so keep it on the detached tmpfs and sparse. + */ + fd_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_file < 0) goto out; - } - if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_INFO_COREDUMP not set in mask\n"); + if (!read_coredump_req(fd_coredump, &req)) goto out; - } - if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_COREDUMPED not set in coredump_mask\n"); + if (!check_coredump_req(&req)) goto out; - } - /* Verify coredump_signal is available and correct */ - if (!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: PIDFD_INFO_COREDUMP_SIGNAL not set in mask\n"); + if (!send_coredump_ack(fd_coredump, &req, ack_mask, 0)) goto out; - } - if (info.coredump_signal != SIGABRT) { - fprintf(stderr, "socket_coredump_signal_sigabrt: coredump_signal=%d, expected SIGABRT=%d\n", - info.coredump_signal, SIGABRT); + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) goto out; - } - if (info.coredump_code != SI_TKILL) { - fprintf(stderr, "socket_coredump_signal_sigabrt: coredump_code=%d, expected SI_TKILL=%d\n", - info.coredump_code, SI_TKILL); + ret = recv_coredump_records(fd_coredump, fd_file, &size, &is_truncated, + kill_peer ? fd_peer_pidfd : -1); + if (ret < 0) goto out; - } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: read_coredump_req failed\n"); + if (write_nointr(pipefds[1], &ret, sizeof(ret)) != sizeof(ret)) goto out; - } - - if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_REJECT | COREDUMP_WAIT, 0)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: send_coredump_ack failed\n"); + if (write_nointr(pipefds[1], &size, sizeof(size)) != sizeof(size)) goto out; - } - - if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { - fprintf(stderr, "socket_coredump_signal_sigabrt: read_marker COREDUMP_MARK_REQACK failed\n"); + if (write_nointr(pipefds[1], &is_truncated, + sizeof(is_truncated)) != sizeof(is_truncated)) goto out; - } exit_code = EXIT_SUCCESS; - fprintf(stderr, "socket_coredump_signal_sigabrt: completed successfully\n"); out: + close(pipefds[1]); + if (fd_file >= 0) + close(fd_file); if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); if (fd_coredump >= 0) @@ -1266,152 +1490,209 @@ TEST_F(coredump, socket_coredump_signal_sigabrt) self->pid_coredump_server = pid_coredump_server; EXPECT_EQ(close(ipc_sockets[1]), 0); + EXPECT_EQ(close(pipefds[1]), 0); ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); EXPECT_EQ(close(ipc_sockets[0]), 0); pid = fork(); ASSERT_GE(pid, 0); if (pid == 0) - abort(); + crashing_child_sparse(SPARSE_MAPPING_SIZE); pidfd = sys_pidfd_open(pid, 0); ASSERT_GE(pidfd, 0); waitpid(pid, &status, 0); ASSERT_TRUE(WIFSIGNALED(status)); - ASSERT_EQ(WTERMSIG(status), SIGABRT); - ASSERT_TRUE(get_pidfd_info(pidfd, &info)); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP)); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_SIGNAL)); - ASSERT_EQ(info.coredump_signal, SIGABRT); - ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP_CODE)); - ASSERT_EQ(info.coredump_code, SI_TKILL); + ASSERT_EQ(read_nointr(pipefds[0], received, sizeof(*received)), + sizeof(*received)); + ASSERT_EQ(read_nointr(pipefds[0], coredump_size, sizeof(*coredump_size)), + sizeof(*coredump_size)); + ASSERT_EQ(read_nointr(pipefds[0], &truncated, sizeof(truncated)), + sizeof(truncated)); + EXPECT_EQ(close(pipefds[0]), 0); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); + + if (kill_peer) { + /* The kernel gave up partway, so no end record closed the stream. */ + ASSERT_TRUE(truncated); + ASSERT_FALSE(WCOREDUMP(status)); + ASSERT_LT(*coredump_size, (off_t)SPARSE_MAPPING_SIZE); + return; + } + + ASSERT_FALSE(truncated); + ASSERT_TRUE(WCOREDUMP(status)); + + ASSERT_TRUE(get_pidfd_info(pidfd, &info)); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + + /* The mapping is in the coredump, holes included. */ + ASSERT_GT(*coredump_size, (off_t)SPARSE_MAPPING_SIZE); } -TEST_F_TIMEOUT(coredump, socket_multiple_crashing_coredumps, 500) +/* + * A mapping that has been written to is dumped whole, including the parts + * of it that were never faulted in. With COREDUMP_SPARSE the holes stay + * off the wire. + */ +TEST_F(coredump, socket_request_sparse_hole) { - int pidfd[NUM_CRASHING_COREDUMPS], status[NUM_CRASHING_COREDUMPS]; - pid_t pid[NUM_CRASHING_COREDUMPS], pid_coredump_server; + off_t coredump_size = 0; + ssize_t received = 0; + + check_record_dump(_metadata, self, + COREDUMP_KERNEL | COREDUMP_RECORDS | + COREDUMP_SPARSE | COREDUMP_WAIT, + false, &received, &coredump_size); + + /* The holes didn't have to go over the socket. */ + ASSERT_LT(received, coredump_size / 8); +} + +/* + * COREDUMP_RECORDS alone splits the stream into records but elides + * nothing: the holes cross the socket as data records. + */ +TEST_F(coredump, socket_request_records_hole) +{ + off_t coredump_size = 0; + ssize_t received = 0; + + check_record_dump(_metadata, self, + COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_WAIT, + false, &received, &coredump_size); + + /* Records alone elide nothing, so everything crossed the socket. */ + ASSERT_GT(received, coredump_size); +} + +/* + * A coredump the kernel gives up on halfway still ends in an end record, + * and that record says the coredump is incomplete. COREDUMP_SPARSE is left + * out on purpose: the holes have to cross the socket so the coredump is + * far larger than the socket buffer and the kernel is still writing it + * when the kill lands. + */ +TEST_F(coredump, socket_request_records_truncated) +{ + off_t coredump_size = 0; + ssize_t received = 0; + + check_record_dump(_metadata, self, + COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_WAIT, + true, &received, &coredump_size); + + /* The end record crossed the socket even though the task was killed. */ + ASSERT_GT(received, 0); +} + +/* + * A coredump server that uploads to a blob store can't upload a sparse + * file. It doesn't have to: it streams the data records into the object + * as they arrive, leaves the holes out, and uploads the corrected + * program header table last. What it ends up with is an ordinary ELF + * core file that describes the same memory as the coredump the records + * came from, minus the holes. + */ +TEST_F(coredump, socket_request_sparse_blob_upload) +{ + int fd_core_file, pidfd, status; + pid_t pid, pid_coredump_server; struct pidfd_info info = {}; + off_t coredump_size = 0; + ssize_t received = 0; int ipc_sockets[2]; + int pipefds[2]; + struct stat st; char c; - ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); - ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + ASSERT_EQ(pipe(pipefds), 0); + ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1, fd_core_file = -1; + int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_object = -1, fd_reference = -1; int exit_code = EXIT_FAILURE; struct coredump_req req = {}; + off_t size = 0; + ssize_t ret; close(ipc_sockets[0]); + close(pipefds[0]); + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); - if (fd_server < 0) { - fprintf(stderr, "Failed to create and listen on unix socket\n"); + if (fd_server < 0) goto out; - } - if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "Failed to notify parent via ipc socket\n"); + if (write_nointr(ipc_sockets[1], "1", 1) < 0) goto out; - } - close(ipc_sockets[1]); - - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - fprintf(stderr, "accept4 failed: %m\n"); - goto out; - } - - fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "get_peer_pidfd failed for fd %d: %m\n", fd_coredump); - goto out; - } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "get_pidfd_info failed for fd %d\n", fd_peer_pidfd); - goto out; - } + close(ipc_sockets[1]); - if (!(info.mask & PIDFD_INFO_COREDUMP)) { - fprintf(stderr, "pidfd info missing PIDFD_INFO_COREDUMP for fd %d\n", fd_peer_pidfd); - goto out; - } - if (!(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "pidfd info missing PIDFD_COREDUMPED for fd %d\n", fd_peer_pidfd); - goto out; - } + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) + goto out; - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "read_coredump_req failed for fd %d\n", fd_coredump); - goto out; - } + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) + goto out; - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "check_coredump_req failed for fd %d\n", fd_coredump); - goto out; - } + /* The object is a plain file. It never sees a hole. */ + fd_object = open("/tmp/coredump.file", + O_RDWR | O_CREAT | O_TRUNC | O_CLOEXEC, 0600); + if (fd_object < 0) + goto out; - if (!send_coredump_ack(fd_coredump, &req, - COREDUMP_KERNEL | COREDUMP_WAIT, 0)) { - fprintf(stderr, "send_coredump_ack failed for fd %d\n", fd_coredump); - goto out; - } + /* + * The coredump with its holes still in it is bigger than + * the mapping the child made, so keep it on the detached + * tmpfs and sparse. + */ + fd_reference = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_reference < 0) + goto out; - if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { - fprintf(stderr, "read_marker failed for fd %d\n", fd_coredump); - goto out; - } + if (!read_coredump_req(fd_coredump, &req)) + goto out; - fd_core_file = open_coredump_tmpfile(self->fd_tmpfs_detached); - if (fd_core_file < 0) { - fprintf(stderr, "%m - open_coredump_tmpfile failed for fd %d\n", fd_coredump); - goto out; - } + if (!check_coredump_req(&req)) + goto out; - for (;;) { - char buffer[4096]; - ssize_t bytes_read, bytes_write; + if (!send_coredump_ack(fd_coredump, &req, + COREDUMP_KERNEL | COREDUMP_RECORDS | + COREDUMP_SPARSE | COREDUMP_WAIT, 0)) + goto out; - bytes_read = read(fd_coredump, buffer, sizeof(buffer)); - if (bytes_read < 0) { - fprintf(stderr, "read failed for fd %d: %m\n", fd_coredump); - goto out; - } + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) + goto out; - if (bytes_read == 0) - break; + ret = recv_coredump_compact(fd_coredump, fd_object, + fd_reference, &size); + if (ret < 0) + goto out; - bytes_write = write(fd_core_file, buffer, bytes_read); - if (bytes_read != bytes_write) { - if (bytes_write < 0 && errno == ENOSPC) - continue; - fprintf(stderr, "write failed for fd %d: %m\n", fd_core_file); - goto out; - } - } + if (check_compact_coredump(fd_object, fd_reference)) + goto out; - close(fd_core_file); - close(fd_peer_pidfd); - close(fd_coredump); - fd_peer_pidfd = -1; - fd_coredump = -1; - } + if (write_nointr(pipefds[1], &ret, sizeof(ret)) != sizeof(ret)) + goto out; + if (write_nointr(pipefds[1], &size, sizeof(size)) != sizeof(size)) + goto out; exit_code = EXIT_SUCCESS; out: - if (fd_core_file >= 0) - close(fd_core_file); + close(pipefds[1]); + if (fd_reference >= 0) + close(fd_reference); + if (fd_object >= 0) + close(fd_object); if (fd_peer_pidfd >= 0) close(fd_peer_pidfd); if (fd_coredump >= 0) @@ -1423,172 +1704,912 @@ TEST_F_TIMEOUT(coredump, socket_multiple_crashing_coredumps, 500) self->pid_coredump_server = pid_coredump_server; EXPECT_EQ(close(ipc_sockets[1]), 0); + EXPECT_EQ(close(pipefds[1]), 0); ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); EXPECT_EQ(close(ipc_sockets[0]), 0); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - pid[i] = fork(); - ASSERT_GE(pid[i], 0); - if (pid[i] == 0) - crashing_child(); - pidfd[i] = sys_pidfd_open(pid[i], 0); - ASSERT_GE(pidfd[i], 0); - } + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_sparse(SPARSE_MAPPING_SIZE); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - waitpid(pid[i], &status[i], 0); - ASSERT_TRUE(WIFSIGNALED(status[i])); - ASSERT_TRUE(WCOREDUMP(status[i])); - } + pidfd = sys_pidfd_open(pid, 0); + ASSERT_GE(pidfd, 0); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - info.mask = PIDFD_INFO_EXIT | PIDFD_INFO_COREDUMP; - ASSERT_EQ(ioctl(pidfd[i], PIDFD_GET_INFO, &info), 0); - ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); - ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); - } + waitpid(pid, &status, 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_TRUE(WCOREDUMP(status)); + + ASSERT_EQ(read_nointr(pipefds[0], &received, sizeof(received)), + sizeof(received)); + ASSERT_EQ(read_nointr(pipefds[0], &coredump_size, sizeof(coredump_size)), + sizeof(coredump_size)); + EXPECT_EQ(close(pipefds[0]), 0); + + ASSERT_TRUE(get_pidfd_info(pidfd, &info)); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); + + /* The mapping is in the coredump, holes included. */ + ASSERT_GT(coredump_size, (off_t)SPARSE_MAPPING_SIZE); + + /* The object isn't sparse and doesn't carry them. */ + ASSERT_EQ(stat("/tmp/coredump.file", &st), 0); + ASSERT_LT(st.st_size, coredump_size / 8); + + /* And a debugger still sees an ordinary ELF core file. */ + fd_core_file = open("/tmp/coredump.file", O_RDONLY | O_CLOEXEC); + ASSERT_GE(fd_core_file, 0); + ASSERT_TRUE(is_elf_core(fd_core_file)); + EXPECT_EQ(close(fd_core_file), 0); } -TEST_F_TIMEOUT(coredump, socket_multiple_crashing_coredumps_epoll_workers, 500) +/* COREDUMP_RECORDS applies to a coredump the kernel writes, nothing else. */ +TEST_F(coredump, socket_request_records_without_kernel) { - int pidfd[NUM_CRASHING_COREDUMPS], status[NUM_CRASHING_COREDUMPS]; - pid_t pid[NUM_CRASHING_COREDUMPS], pid_coredump_server, worker_pids[NUM_CRASHING_COREDUMPS]; + check_conflicting_ack(_metadata, self, COREDUMP_USERSPACE | COREDUMP_RECORDS); +} + +/* A zero record can't exist outside a record stream. */ +TEST_F(coredump, socket_request_sparse_without_records) +{ + check_conflicting_ack(_metadata, self, COREDUMP_KERNEL | COREDUMP_SPARSE); +} + +/* What the server reports back about the coredump it decided to take. */ +struct stream_choice { + bool sparse; + ssize_t received; + off_t size; + ssize_t vm_size; +}; + +/* + * The kernel blocks in the coredump request until the ack arrives, so a + * coredump server gets to look at the task before it commits to a + * stream. Take the record stream only for a task whose mappings are + * worth it and the plain byte stream for everything else. + */ +static void check_stream_choice(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, bool big, + struct stream_choice *choice) +{ + int pidfd, status; + pid_t pid, pid_coredump_server; struct pidfd_info info = {}; int ipc_sockets[2]; + int pipefds[2]; char c; - ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + ASSERT_EQ(pipe(pipefds), 0); + ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); pid_coredump_server = fork(); ASSERT_GE(pid_coredump_server, 0); if (pid_coredump_server == 0) { - int fd_server = -1, exit_code = EXIT_FAILURE, n_conns = 0; - fd_server = -1; - exit_code = EXIT_FAILURE; - n_conns = 0; + int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_file = -1; + int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; + struct stream_choice got = {}; + __u64 mask; + close(ipc_sockets[0]); + close(pipefds[0]); + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); - if (fd_server < 0) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: create_and_listen_unix_socket failed: %m\n"); + if (fd_server < 0) goto out; - } - if (write_nointr(ipc_sockets[1], "1", 1) < 0) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: write_nointr to ipc socket failed: %m\n"); + if (write_nointr(ipc_sockets[1], "1", 1) < 0) + goto out; + + close(ipc_sockets[1]); + + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) + goto out; + + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) + goto out; + + /* + * The reassembled coredump is bigger than the mapping the + * child made, so keep it on the detached tmpfs and sparse. + */ + fd_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_file < 0) + goto out; + + if (!read_coredump_req(fd_coredump, &req)) + goto out; + + if (!check_coredump_req(&req)) + goto out; + + /* + * Nothing is on the wire yet and the kernel is waiting for + * the ack, so there is all the time in the world to look at + * the task and decide what to ask it for. + */ + got.vm_size = peer_vm_size(fd_peer_pidfd); + if (got.vm_size < 0) + goto out; + got.sparse = got.vm_size >= SPARSE_STREAM_THRESHOLD; + + fprintf(stderr, "Peer maps %zd bytes, asking for %s\n", + got.vm_size, + got.sparse ? "a sparse record stream" : "a byte stream"); + + mask = COREDUMP_KERNEL | COREDUMP_WAIT; + if (got.sparse) + mask |= COREDUMP_RECORDS | COREDUMP_SPARSE; + + if (!send_coredump_ack(fd_coredump, &req, mask, 0)) + goto out; + + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) goto out; + + if (got.sparse) { + got.received = recv_coredump_records(fd_coredump, fd_file, + &got.size, NULL, -1); + } else { + got.received = recv_coredump_bytes(fd_coredump, fd_file); + got.size = got.received; } + if (got.received < 0) + goto out; + + /* Either way a debugger has to see an ordinary core file. */ + if (!is_elf_core(fd_file)) + goto out; + + if (write_nointr(pipefds[1], &got, sizeof(got)) != sizeof(got)) + goto out; + + exit_code = EXIT_SUCCESS; +out: + close(pipefds[1]); + if (fd_file >= 0) + close(fd_file); + if (fd_peer_pidfd >= 0) + close(fd_peer_pidfd); + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_server >= 0) + close(fd_server); + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); + EXPECT_EQ(close(pipefds[1]), 0); + ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); + EXPECT_EQ(close(ipc_sockets[0]), 0); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_sparse(big ? SPARSE_MAPPING_SIZE : PAGE_SIZE); + + pidfd = sys_pidfd_open(pid, 0); + ASSERT_GE(pidfd, 0); + + waitpid(pid, &status, 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_TRUE(WCOREDUMP(status)); + + ASSERT_EQ(read_nointr(pipefds[0], choice, sizeof(*choice)), + sizeof(*choice)); + EXPECT_EQ(close(pipefds[0]), 0); + + ASSERT_TRUE(get_pidfd_info(pidfd, &info)); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); + + wait_and_check_coredump_server(pid_coredump_server, _metadata, self); +} + +/* A task with little mapped isn't worth a record stream. */ +TEST_F(coredump, socket_request_stream_choice_small) +{ + struct stream_choice choice = {}; + + check_stream_choice(_metadata, self, false, &choice); + + ASSERT_LT(choice.vm_size, (ssize_t)SPARSE_STREAM_THRESHOLD); + ASSERT_FALSE(choice.sparse); + ASSERT_GT(choice.received, 0); +} + +/* A task sitting on a big mapping is. */ +TEST_F(coredump, socket_request_stream_choice_large) +{ + struct stream_choice choice = {}; + + check_stream_choice(_metadata, self, true, &choice); + + ASSERT_GE(choice.vm_size, (ssize_t)SPARSE_STREAM_THRESHOLD); + ASSERT_TRUE(choice.sparse); + ASSERT_GT(choice.size, (off_t)SPARSE_MAPPING_SIZE); + + /* The holes didn't have to go over the socket. */ + ASSERT_LT(choice.received, choice.size / 8); +} + +/* What a coredump server was built with. */ +struct server_build { + /* sizeof(struct coredump_req) and sizeof(struct coredump_ack) back then. */ + size_t req_size; + size_t ack_size; + /* The features it raises if the kernel offers them. */ + __u64 wants; + /* Its policy: what it drops from and adds to the task's selection. */ + __u64 drop; + __u64 add; +}; + +/* A server from when the structs were first published: kernel-written dumps. */ +static const struct server_build server_build_ver0 = { + .req_size = COREDUMP_REQ_SIZE_VER0, + .ack_size = COREDUMP_ACK_SIZE_VER0, + .wants = COREDUMP_KERNEL, +}; + +/* A server built against this header: no shared memory, always the ELF headers. */ +static const struct server_build server_build_ver1 = { + .req_size = sizeof(struct coredump_req), + .ack_size = sizeof(struct coredump_ack), + .wants = COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES, + .drop = COREDUMP_MEMORY_ANON_SHARED | COREDUMP_MEMORY_FILE_SHARED, + .add = COREDUMP_MEMORY_ELF_HEADERS, +}; + +/* + * Build the ack the way a server does: from what the kernel offers, what + * this build implements, and what fits in the ack the kernel accepts. + * Fields the build never read are zero and never consulted. + */ +static void negotiate(const struct coredump_req *req, + const struct server_build *build, + struct coredump_ack *ack) +{ + __u64 offered = req->mask & build->wants; + + memset(ack, 0, sizeof(*ack)); + ack->size = build->ack_size < req->size_ack ? build->ack_size : req->size_ack; + /* These builds only ever have the kernel write the coredump. */ + ack->mask = COREDUMP_KERNEL; + + /* Sparse needs records, records need the kernel to write. */ + if (offered & COREDUMP_RECORDS) { + ack->mask |= COREDUMP_RECORDS; + if (offered & COREDUMP_SPARSE) + ack->mask |= COREDUMP_SPARSE; + } + + /* The memory types need an ack that carries them. */ + if ((offered & COREDUMP_MEMORY_TYPES) && ack->size >= COREDUMP_ACK_SIZE_VER1) { + ack->mask |= COREDUMP_MEMORY_TYPES; + /* Start from the task's selection; only advertised types pass. */ + ack->memory_types = (req->memory_types & ~build->drop) | build->add; + ack->memory_types &= req->memory_types_mask; + } +} + +/* What a memory types test asks of the kernel and what it expects back. */ +struct memory_choice { + /* Memory types the crashing child selects, or FILTER_TASK_INHERIT. */ + __u64 task_filter; + /* Negotiate the ack as this server build, NULL to send it as given. */ + const struct server_build *build; + /* The ack, or what the negotiation must arrive at. */ + __u64 mask; + __u64 memory_types; + size_t size_ack; + /* The shared mapping is in the coredump with all of its memory. */ + bool shared_dumped; + /* No memory at all. Pull a page from /proc//mem instead. */ + bool skeleton; +}; + +/* A skeleton still carries the vdso and friends, nothing bigger. */ +#define SKELETON_DATA_PAGES 16 + +/* + * The crashing child maps shared anonymous memory and tells the server + * where. The server acks with @choice and checks whether that mapping's + * segment in the coredump carries its memory. + */ +static void check_memory_dump(struct __test_metadata *const _metadata, + FIXTURE_DATA(coredump) *self, + const struct memory_choice *choice) +{ + int pidfd, status; + pid_t pid, pid_coredump_server; + struct pidfd_info info = {}; + int ipc_sockets[2]; + int addr_pipe[2]; + char c; + + ASSERT_EQ(socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets), 0); + ASSERT_EQ(pipe(addr_pipe), 0); + ASSERT_TRUE(set_core_pattern("@@/tmp/coredump.socket")); + + pid_coredump_server = fork(); + ASSERT_GE(pid_coredump_server, 0); + if (pid_coredump_server == 0) { + int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_file = -1; + int exit_code = EXIT_FAILURE; + struct coredump_req req = {}; + struct coredump_ack ack = { + .size = choice->size_ack, + .mask = choice->mask, + .memory_types = choice->memory_types, + }; + /* How much of the request this server reads. */ + size_t req_size = choice->build ? choice->build->req_size : sizeof(req); + __u64 task_filter; + ElfW(Phdr) segment; + ssize_t received; + off_t size; + char *addr; + + close(ipc_sockets[0]); + close(addr_pipe[1]); + + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); + if (fd_server < 0) + goto out; + + if (write_nointr(ipc_sockets[1], "1", 1) < 0) + goto out; + close(ipc_sockets[1]); - while (n_conns < NUM_CRASHING_COREDUMPS) { - int fd_coredump = -1, fd_peer_pidfd = -1, fd_core_file = -1; - struct coredump_req req = {}; - fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); - if (fd_coredump < 0) { - if (errno == EAGAIN || errno == EWOULDBLOCK) - continue; - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: accept4 failed: %m\n"); - goto out; - } - fd_peer_pidfd = get_peer_pidfd(fd_coredump); - if (fd_peer_pidfd < 0) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: get_peer_pidfd failed\n"); - goto out; - } - if (!get_pidfd_info(fd_peer_pidfd, &info)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: get_pidfd_info failed\n"); + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) + goto out; + + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) + goto out; + + fd_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_file < 0) + goto out; + + if (!read_coredump_req_sized(fd_coredump, &req, req_size)) + goto out; + + if (!peer_coredump_filter(fd_peer_pidfd, &task_filter)) + goto out; + + /* A build from before the memory types never read that far. */ + if (req_size >= COREDUMP_REQ_SIZE_VER1) { + if (!check_coredump_req(&req)) goto out; - } - if (!(info.mask & PIDFD_INFO_COREDUMP) || !(info.coredump_mask & PIDFD_COREDUMPED)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: missing PIDFD_INFO_COREDUMP or PIDFD_COREDUMPED\n"); + + /* The request reports the memory types the task selected. */ + if (req.memory_types != task_filter) { + fprintf(stderr, "Request reports 0x%llx, task selected 0x%llx\n", + (unsigned long long)req.memory_types, + (unsigned long long)task_filter); goto out; } - if (!read_coredump_req(fd_coredump, &req)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: read_coredump_req failed\n"); + } + + if (choice->task_filter != FILTER_TASK_INHERIT && + task_filter != choice->task_filter) { + fprintf(stderr, "Task selected 0x%llx, child asked for 0x%llx\n", + (unsigned long long)task_filter, + (unsigned long long)choice->task_filter); + goto out; + } + + /* The child sent the address of its mapping before it crashed. */ + if (read_nointr(addr_pipe[0], &addr, sizeof(addr)) != sizeof(addr)) + goto out; + + /* A server build negotiates its ack and must arrive at the choice. */ + if (choice->build) { + negotiate(&req, choice->build, &ack); + + if (ack.size != choice->size_ack || ack.mask != choice->mask || + ack.memory_types != choice->memory_types) { + fprintf(stderr, + "Negotiated %u bytes, mask 0x%llx, types 0x%llx\n", + ack.size, (unsigned long long)ack.mask, + (unsigned long long)ack.memory_types); goto out; } - if (!check_coredump_req(&req, COREDUMP_ACK_SIZE_VER0, - COREDUMP_KERNEL | COREDUMP_USERSPACE | - COREDUMP_REJECT | COREDUMP_WAIT)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: check_coredump_req failed\n"); + } + + if (!send_coredump_ack_types(fd_coredump, &req, ack.mask, + ack.memory_types, ack.size)) + goto out; + + if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) + goto out; + + if (ack.mask & COREDUMP_RECORDS) + received = recv_coredump_records(fd_coredump, fd_file, + &size, NULL, -1); + else + received = recv_coredump_bytes(fd_coredump, fd_file); + if (received < 0) + goto out; + + if (!is_elf_core(fd_file)) + goto out; + + /* A dump ending in holes or empty segments must still be whole. */ + if (!check_coredump_extent(fd_file)) + goto out; + + if (!find_coredump_segment(fd_file, (__u64)(uintptr_t)addr, &segment)) + goto out; + + if (segment.p_memsz != MEMORY_MAPPING_SIZE) { + fprintf(stderr, "Segment spans %llu bytes, the mapping %u\n", + (unsigned long long)segment.p_memsz, + MEMORY_MAPPING_SIZE); + goto out; + } + + if (segment.p_filesz != (choice->shared_dumped ? segment.p_memsz : 0)) { + fprintf(stderr, "Segment carries %llu bytes, expected %s of them\n", + (unsigned long long)segment.p_filesz, + choice->shared_dumped ? "all" : "none"); + goto out; + } + + if (choice->skeleton) { + __u64 data, notes, data_max; + char buf[PAGE_SIZE]; + + if (!sum_coredump_segments(fd_file, &data, ¬es)) goto out; - } - if (!send_coredump_ack(fd_coredump, &req, COREDUMP_KERNEL | COREDUMP_WAIT, 0)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: send_coredump_ack failed\n"); + + data_max = SKELETON_DATA_PAGES * sysconf(_SC_PAGESIZE); + if (!notes || data > data_max) { + fprintf(stderr, "Skeleton has %llu note and %llu memory bytes\n", + (unsigned long long)notes, + (unsigned long long)data); goto out; } - if (!read_marker(fd_coredump, COREDUMP_MARK_REQACK)) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: read_marker failed\n"); + + /* The task is parked in COREDUMP_WAIT with its memory. */ + if (peer_read_mem(fd_peer_pidfd, (__u64)(uintptr_t)addr, + buf, sizeof(buf)) != sizeof(buf)) goto out; - } - fd_core_file = open_coredump_tmpfile(self->fd_tmpfs_detached); - if (fd_core_file < 0) { - fprintf(stderr, "socket_multiple_crashing_coredumps_epoll_workers: open_coredump_tmpfile failed: %m\n"); + + if (buf[0] != 'x') { + fprintf(stderr, "Pulled memory lacks the child's mark\n"); goto out; } - pid_t worker = fork(); - if (worker == 0) { - close(fd_server); - process_coredump_worker(fd_coredump, fd_peer_pidfd, fd_core_file); - } - worker_pids[n_conns] = worker; - if (fd_coredump >= 0) - close(fd_coredump); - if (fd_peer_pidfd >= 0) - close(fd_peer_pidfd); - if (fd_core_file >= 0) - close(fd_core_file); - n_conns++; + + fprintf(stderr, "Skeleton of %zd bytes, pulled %zu bytes of memory\n", + received, sizeof(buf)); } + exit_code = EXIT_SUCCESS; out: + close(addr_pipe[0]); + if (fd_file >= 0) + close(fd_file); + if (fd_peer_pidfd >= 0) + close(fd_peer_pidfd); + if (fd_coredump >= 0) + close(fd_coredump); if (fd_server >= 0) close(fd_server); - - // Reap all worker processes - for (int i = 0; i < n_conns; i++) { - int wstatus; - if (waitpid(worker_pids[i], &wstatus, 0) < 0) { - fprintf(stderr, "Failed to wait for worker %d: %m\n", worker_pids[i]); - } else if (WIFEXITED(wstatus) && WEXITSTATUS(wstatus) != EXIT_SUCCESS) { - fprintf(stderr, "Worker %d exited with error code %d\n", worker_pids[i], WEXITSTATUS(wstatus)); - exit_code = EXIT_FAILURE; - } - } - _exit(exit_code); } self->pid_coredump_server = pid_coredump_server; EXPECT_EQ(close(ipc_sockets[1]), 0); + EXPECT_EQ(close(addr_pipe[0]), 0); ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); EXPECT_EQ(close(ipc_sockets[0]), 0); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - pid[i] = fork(); - ASSERT_GE(pid[i], 0); - if (pid[i] == 0) - crashing_child(); - pidfd[i] = sys_pidfd_open(pid[i], 0); - ASSERT_GE(pidfd[i], 0); - } + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_memory(choice->task_filter, addr_pipe[1]); + EXPECT_EQ(close(addr_pipe[1]), 0); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - ASSERT_GE(waitpid(pid[i], &status[i], 0), 0); - ASSERT_TRUE(WIFSIGNALED(status[i])); - ASSERT_TRUE(WCOREDUMP(status[i])); - } + pidfd = sys_pidfd_open(pid, 0); + ASSERT_GE(pidfd, 0); - for (int i = 0; i < NUM_CRASHING_COREDUMPS; i++) { - info.mask = PIDFD_INFO_EXIT | PIDFD_INFO_COREDUMP; - ASSERT_EQ(ioctl(pidfd[i], PIDFD_GET_INFO, &info), 0); - ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); - ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); - } + waitpid(pid, &status, 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_TRUE(WCOREDUMP(status)); + + ASSERT_TRUE(get_pidfd_info(pidfd, &info)); + ASSERT_GT((info.mask & PIDFD_INFO_COREDUMP), 0); + ASSERT_GT((info.coredump_mask & PIDFD_COREDUMPED), 0); wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } +/* Without COREDUMP_MEMORY_TYPES the task's own selection decides. */ +TEST_F(coredump, socket_request_memory_types_task_includes) +{ + struct memory_choice choice = { + .task_filter = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .mask = COREDUMP_KERNEL, + .shared_dumped = true, + }; + + check_memory_dump(_metadata, self, &choice); +} + +TEST_F(coredump, socket_request_memory_types_task_excludes) +{ + struct memory_choice choice = { + .task_filter = 0, + .mask = COREDUMP_KERNEL, + .shared_dumped = false, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* The server drops a memory type the task would have dumped. */ +TEST_F(coredump, socket_request_memory_types_restricts) +{ + struct memory_choice choice = { + .task_filter = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .mask = COREDUMP_KERNEL | COREDUMP_MEMORY_TYPES, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE, + .shared_dumped = false, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* The server adds a memory type the task had excluded. */ +TEST_F(coredump, socket_request_memory_types_widens) +{ + struct memory_choice choice = { + .task_filter = 0, + .mask = COREDUMP_KERNEL | COREDUMP_MEMORY_TYPES, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .shared_dumped = true, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* The memory types decide what goes into a record stream just the same. */ +TEST_F(coredump, socket_request_memory_types_records) +{ + struct memory_choice choice = { + .task_filter = FILTER_TASK_INHERIT, + .mask = COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE, + .shared_dumped = false, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* + * An empty selection leaves a skeleton: every program header and every note + * but no memory. A server that wants to pick the memory itself reads it + * from /proc//mem while the task waits for it to finish. + */ +TEST_F(coredump, socket_request_memory_types_skeleton) +{ + struct memory_choice choice = { + .task_filter = FILTER_TASK_INHERIT, + .mask = COREDUMP_KERNEL | COREDUMP_WAIT | COREDUMP_MEMORY_TYPES, + .memory_types = 0, + .shared_dumped = false, + .skeleton = true, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* A memory type the kernel didn't advertise in memory_types_mask. */ +TEST_F(coredump, socket_request_memory_types_unknown_bit) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .mask = COREDUMP_KERNEL | COREDUMP_MEMORY_TYPES, + .memory_types = 1ULL << 63, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_UNSUPPORTED, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* The memory types must be zero unless COREDUMP_MEMORY_TYPES is raised. */ +TEST_F(coredump, socket_request_memory_types_stale_field) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .mask = COREDUMP_KERNEL, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_UNSUPPORTED, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* COREDUMP_MEMORY_TYPES needs an ack that has the memory types. */ +TEST_F(coredump, socket_request_memory_types_short_ack) +{ + struct refused_ack refused = { + .ack = { + .size = COREDUMP_ACK_SIZE_VER0, + .mask = COREDUMP_KERNEL | COREDUMP_MEMORY_TYPES, + }, + .bytes = COREDUMP_ACK_SIZE_VER0, + .mark = COREDUMP_MARK_MINSIZE, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* The memory types select what the kernel writes, nothing else. */ +TEST_F(coredump, socket_request_memory_types_without_kernel) +{ + check_conflicting_ack(_metadata, self, COREDUMP_USERSPACE | COREDUMP_MEMORY_TYPES); +} + +/* + * A server built with the first structs reads the request it knows, + * discards the rest and acks with the ack it knows. It raises nothing + * it wasn't built for and the kernel dumps what the task selected. + */ +TEST_F(coredump, socket_request_negotiate_ver0) +{ + struct memory_choice choice = { + .task_filter = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .build = &server_build_ver0, + .mask = COREDUMP_KERNEL, + .memory_types = 0, + .size_ack = COREDUMP_ACK_SIZE_VER0, + .shared_dumped = true, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* + * A server built against this header takes every feature the kernel + * offers, drops shared memory from what the task selected and adds the + * ELF headers. + */ +TEST_F(coredump, socket_request_negotiate_ver1) +{ + struct memory_choice choice = { + .task_filter = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .build = &server_build_ver1, + .mask = COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ELF_HEADERS, + .size_ack = COREDUMP_ACK_SIZE_VER1, + .shared_dumped = false, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* An ack that picks none of KERNEL, USERSPACE and REJECT. */ +TEST_F(coredump, socket_request_no_mode) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .mask = COREDUMP_WAIT, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_CONFLICTING, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* @spare must be zero, like every field that isn't in use. */ +TEST_F(coredump, socket_request_spare) +{ + struct refused_ack refused = { + .ack = { + .size = sizeof(struct coredump_ack), + .spare = 1, + .mask = COREDUMP_KERNEL, + }, + .bytes = sizeof(struct coredump_ack), + .mark = COREDUMP_MARK_UNSUPPORTED, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* An ack size is a byte count. One that ends inside a field is valid. */ +#define ACK_SIZE_BETWEEN (COREDUMP_ACK_SIZE_VER0 + sizeof(__u32)) + +/* Any size from VER0 up to what the kernel accepts works without memory types. */ +TEST_F(coredump, socket_request_ack_size_between) +{ + struct memory_choice choice = { + .task_filter = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .mask = COREDUMP_KERNEL, + .size_ack = ACK_SIZE_BETWEEN, + .shared_dumped = true, + }; + + check_memory_dump(_metadata, self, &choice); +} + +/* The memory types need the whole field, not the part that happens to fit. */ +TEST_F(coredump, socket_request_memory_types_ack_size_between) +{ + struct refused_ack refused = { + .ack = { + .size = ACK_SIZE_BETWEEN, + .mask = COREDUMP_KERNEL | COREDUMP_MEMORY_TYPES, + }, + .bytes = ACK_SIZE_BETWEEN, + .mark = COREDUMP_MARK_MINSIZE, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* A server that hangs up without acking gets no marker and no coredump. */ +TEST_F(coredump, socket_request_server_hangs_up) +{ + struct refused_ack refused = { + .bytes = 0, + .no_marker = true, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* A server that hangs up in the middle of its ack looks the same. */ +TEST_F(coredump, socket_request_ack_truncated) +{ + struct refused_ack refused = { + .ack = { + .size = COREDUMP_ACK_SIZE_VER0, + .mask = COREDUMP_KERNEL, + }, + .bytes = COREDUMP_ACK_SIZE_VER0 / 2, + .no_marker = true, + }; + + check_refused_ack(_metadata, self, &refused); +} + +/* + * The kernels a server built against this header can't meet here: + * negotiate() against their requests, no coredump involved. + */ + +/* The request of a kernel with the first structs and features. */ +static const struct coredump_req req_ver0 = { + .size = COREDUMP_REQ_SIZE_VER0, + .size_ack = COREDUMP_ACK_SIZE_VER0, + .mask = COREDUMP_KERNEL | COREDUMP_USERSPACE | + COREDUMP_REJECT | COREDUMP_WAIT, +}; + +/* The request of this kernel. */ +static const struct coredump_req req_ver1 = { + .size = COREDUMP_REQ_SIZE_VER1, + .size_ack = COREDUMP_ACK_SIZE_VER1, + .mask = COREDUMP_KERNEL | COREDUMP_USERSPACE | + COREDUMP_REJECT | COREDUMP_WAIT | + COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES, + .memory_types = COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ANON_SHARED, + .memory_types_mask = TEST_MEMORY_ALL, +}; + +/* A kernel with the first structs gets the first ack and nothing newer. */ +TEST(negotiate_ver0_kernel) +{ + struct coredump_ack ack; + + negotiate(&req_ver0, &server_build_ver1, &ack); + ASSERT_EQ(ack.size, COREDUMP_ACK_SIZE_VER0); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL); + ASSERT_EQ(ack.memory_types, 0); +} + +/* A kernel with records and sparse but the first structs: both, no types. */ +TEST(negotiate_sparse_kernel) +{ + struct coredump_req req = req_ver0; + struct coredump_ack ack; + + req.mask |= COREDUMP_RECORDS | COREDUMP_SPARSE; + negotiate(&req, &server_build_ver1, &ack); + ASSERT_EQ(ack.size, COREDUMP_ACK_SIZE_VER0); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE); + ASSERT_EQ(ack.memory_types, 0); +} + +/* Records without sparse: sparse isn't raised on its own. */ +TEST(negotiate_records_without_sparse) +{ + struct coredump_req req = req_ver0; + struct coredump_ack ack; + + req.mask |= COREDUMP_RECORDS; + negotiate(&req, &server_build_ver1, &ack); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL | COREDUMP_RECORDS); +} + +/* + * A feature whose ack field lies past what the kernel accepts can't be + * raised. No kernel offers the memory types without the room for them, so a + * request that does stands in for a feature newer than this header. + */ +TEST(negotiate_types_need_room) +{ + struct coredump_req req = req_ver0; + struct coredump_ack ack; + + req.mask |= COREDUMP_MEMORY_TYPES; + negotiate(&req, &server_build_ver1, &ack); + ASSERT_EQ(ack.size, COREDUMP_ACK_SIZE_VER0); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL); + ASSERT_EQ(ack.memory_types, 0); +} + +/* This kernel: the policy applied to the task's selection. */ +TEST(negotiate_ver1_kernel) +{ + struct coredump_ack ack; + + negotiate(&req_ver1, &server_build_ver1, &ack); + ASSERT_EQ(ack.size, COREDUMP_ACK_SIZE_VER1); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES); + ASSERT_EQ(ack.memory_types, COREDUMP_MEMORY_ANON_PRIVATE | + COREDUMP_MEMORY_ELF_HEADERS); +} + +/* A kernel that doesn't know a type the policy adds isn't asked for it. */ +TEST(negotiate_unknown_type) +{ + struct coredump_req req = req_ver1; + struct coredump_ack ack; + + req.memory_types_mask &= ~(__u64)COREDUMP_MEMORY_ELF_HEADERS; + negotiate(&req, &server_build_ver1, &ack); + ASSERT_EQ(ack.mask, COREDUMP_KERNEL | COREDUMP_RECORDS | COREDUMP_SPARSE | + COREDUMP_MEMORY_TYPES); + ASSERT_EQ(ack.memory_types, COREDUMP_MEMORY_ANON_PRIVATE); +} + TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/coredump/coredump_socket_test.c b/tools/testing/selftests/coredump/coredump_socket_test.c index 422728f632ca6c..ec73bb690bbcb0 100644 --- a/tools/testing/selftests/coredump/coredump_socket_test.c +++ b/tools/testing/selftests/coredump/coredump_socket_test.c @@ -592,6 +592,181 @@ TEST_F(coredump, socket_coredump_signal_sigsegv) wait_and_check_coredump_server(pid_coredump_server, _metadata, self); } +static bool check_coredump_info(const struct pidfd_info *info, const char *what) +{ + if (!(info->mask & PIDFD_INFO_COREDUMP)) { + fprintf(stderr, "%s: PIDFD_INFO_COREDUMP not set in mask\n", what); + return false; + } + + if (!(info->coredump_mask & PIDFD_COREDUMPED)) { + fprintf(stderr, "%s: PIDFD_COREDUMPED not set in coredump_mask\n", what); + return false; + } + + if (!(info->mask & PIDFD_INFO_COREDUMP_SIGNAL) || info->coredump_signal != SIGSEGV) { + fprintf(stderr, "%s: coredump_signal=%d, expected SIGSEGV=%d\n", + what, info->coredump_signal, SIGSEGV); + return false; + } + + if (!(info->mask & PIDFD_INFO_COREDUMP_CODE) || info->coredump_code != SEGV_MAPERR) { + fprintf(stderr, "%s: coredump_code=%d, expected SEGV_MAPERR=%d\n", + what, info->coredump_code, SEGV_MAPERR); + return false; + } + + return true; +} + +/* + * Test: PIDFD_INFO_COREDUMP on the dumping thread's pidfd + * + * Crash from a non-leader thread and verify that the pidfd from + * SO_PEERPIDFD_THREAD refers to that thread and reports the coredump + * like the thread-group leader's pidfd from SO_PEERPIDFD does. + */ +TEST_F(coredump, socket_coredump_thread) +{ + int pidfd, ret, status; + pid_t pid, pid_coredump_server; + struct pidfd_info info = {}; + int ipc_sockets[2]; + char c; + + ASSERT_TRUE(set_core_pattern("@/tmp/coredump.socket")); + + ret = socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, ipc_sockets); + ASSERT_EQ(ret, 0); + + pid_coredump_server = fork(); + ASSERT_GE(pid_coredump_server, 0); + if (pid_coredump_server == 0) { + int fd_server = -1, fd_coredump = -1, fd_peer_pidfd = -1; + int fd_thread_pidfd = -1, fd_core_file = -1; + struct pidfd_info thread_info = {}; + int exit_code = EXIT_FAILURE; + + close(ipc_sockets[0]); + + fd_server = create_and_listen_unix_socket("/tmp/coredump.socket"); + if (fd_server < 0) { + fprintf(stderr, "socket_coredump_thread: listen socket failed: %m\n"); + goto out; + } + + if (write_nointr(ipc_sockets[1], "1", 1) < 0) { + fprintf(stderr, "socket_coredump_thread: ipc write failed: %m\n"); + goto out; + } + + close(ipc_sockets[1]); + + fd_coredump = accept4(fd_server, NULL, NULL, SOCK_CLOEXEC); + if (fd_coredump < 0) { + fprintf(stderr, "socket_coredump_thread: accept4 failed: %m\n"); + goto out; + } + + fd_peer_pidfd = get_peer_pidfd(fd_coredump); + if (fd_peer_pidfd < 0) { + fprintf(stderr, "socket_coredump_thread: get_peer_pidfd failed\n"); + goto out; + } + + fd_thread_pidfd = get_peer_pidfd_thread(fd_coredump); + if (fd_thread_pidfd < 0) { + fprintf(stderr, "socket_coredump_thread: get_peer_pidfd_thread failed\n"); + goto out; + } + + if (!get_pidfd_info(fd_peer_pidfd, &info) || + !get_pidfd_info(fd_thread_pidfd, &thread_info)) { + fprintf(stderr, "socket_coredump_thread: get_pidfd_info failed\n"); + goto out; + } + + /* The peer is the thread-group leader, the dumping thread is not. */ + if (info.pid != info.tgid || thread_info.tgid != info.tgid || + thread_info.pid == thread_info.tgid) { + fprintf(stderr, "socket_coredump_thread: unexpected ids %d/%d and %d/%d\n", + info.pid, info.tgid, thread_info.pid, thread_info.tgid); + goto out; + } + + if (!check_coredump_info(&info, "SO_PEERPIDFD") || + !check_coredump_info(&thread_info, "SO_PEERPIDFD_THREAD")) + goto out; + + fd_core_file = open_coredump_tmpfile(self->fd_tmpfs_detached); + if (fd_core_file < 0) { + fprintf(stderr, "socket_coredump_thread: core tmpfile failed: %m\n"); + goto out; + } + + for (;;) { + char buffer[4096]; + ssize_t bytes_read, bytes_write; + + bytes_read = read(fd_coredump, buffer, sizeof(buffer)); + if (bytes_read < 0) { + fprintf(stderr, "socket_coredump_thread: core read failed: %m\n"); + goto out; + } + + if (bytes_read == 0) + break; + + bytes_write = write(fd_core_file, buffer, bytes_read); + if (bytes_read != bytes_write) { + fprintf(stderr, "socket_coredump_thread: core write %zd/%zd: %m\n", + bytes_read, bytes_write); + goto out; + } + } + + exit_code = EXIT_SUCCESS; + fprintf(stderr, "socket_coredump_thread: completed successfully\n"); +out: + if (fd_core_file >= 0) + close(fd_core_file); + if (fd_thread_pidfd >= 0) + close(fd_thread_pidfd); + if (fd_peer_pidfd >= 0) + close(fd_peer_pidfd); + if (fd_coredump >= 0) + close(fd_coredump); + if (fd_server >= 0) + close(fd_server); + _exit(exit_code); + } + self->pid_coredump_server = pid_coredump_server; + + EXPECT_EQ(close(ipc_sockets[1]), 0); + ASSERT_EQ(read_nointr(ipc_sockets[0], &c, 1), 1); + EXPECT_EQ(close(ipc_sockets[0]), 0); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) + crashing_child_thread(); + + pidfd = sys_pidfd_open(pid, 0); + ASSERT_GE(pidfd, 0); + + waitpid(pid, &status, 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_EQ(WTERMSIG(status), SIGSEGV); + ASSERT_TRUE(WCOREDUMP(status)); + + ASSERT_TRUE(get_pidfd_info(pidfd, &info)); + ASSERT_TRUE(!!(info.mask & PIDFD_INFO_COREDUMP)); + ASSERT_TRUE(!!(info.coredump_mask & PIDFD_COREDUMPED)); + ASSERT_EQ(info.coredump_signal, SIGSEGV); + + wait_and_check_coredump_server(pid_coredump_server, _metadata, self); +} + /* * Test: PIDFD_INFO_COREDUMP_SIGNAL via simple socket coredump with SIGABRT * diff --git a/tools/testing/selftests/coredump/coredump_test.h b/tools/testing/selftests/coredump/coredump_test.h index ed47f01fa53c52..1c154f51370081 100644 --- a/tools/testing/selftests/coredump/coredump_test.h +++ b/tools/testing/selftests/coredump/coredump_test.h @@ -3,18 +3,10 @@ #ifndef __COREDUMP_TEST_H #define __COREDUMP_TEST_H -#include -#include -#include - #include "../kselftest_harness.h" -#include "../pidfd/pidfd.h" - -#ifndef PAGE_SIZE -#define PAGE_SIZE 4096 -#endif +#include "coredump_notify_signal.h" -#define NUM_THREAD_SPAWN 128 +#include "coredump_test_helpers.h" /* Coredump fixture */ FIXTURE(coredump) @@ -24,15 +16,6 @@ FIXTURE(coredump) int fd_tmpfs_detached; }; -/* Shared helper function declarations */ -void *do_nothing(void *arg); -void crashing_child(void); -int create_detached_tmpfs(void); -int create_and_listen_unix_socket(const char *path); -bool set_core_pattern(const char *pattern); -int get_peer_pidfd(int fd); -bool get_pidfd_info(int fd_peer_pidfd, struct pidfd_info *info); - /* Inline helper that uses harness types */ static inline void wait_and_check_coredump_server(pid_t pid_coredump_server, struct __test_metadata *const _metadata, @@ -45,15 +28,4 @@ static inline void wait_and_check_coredump_server(pid_t pid_coredump_server, ASSERT_EQ(WEXITSTATUS(status), 0); } -/* Protocol helper function declarations */ -ssize_t recv_marker(int fd); -bool read_marker(int fd, enum coredump_mark mark); -bool read_coredump_req(int fd, struct coredump_req *req); -bool send_coredump_ack(int fd, const struct coredump_req *req, - __u64 mask, size_t size_ack); -bool check_coredump_req(const struct coredump_req *req, size_t min_size, - __u64 required_mask); -int open_coredump_tmpfile(int fd_tmpfs_detached); -void process_coredump_worker(int fd_coredump, int fd_peer_pidfd, int fd_core_file); - #endif /* __COREDUMP_TEST_H */ diff --git a/tools/testing/selftests/coredump/coredump_test_helpers.c b/tools/testing/selftests/coredump/coredump_test_helpers.c index 2a20faf9cb0ad3..072ab86248088a 100644 --- a/tools/testing/selftests/coredump/coredump_test_helpers.c +++ b/tools/testing/selftests/coredump/coredump_test_helpers.c @@ -1,11 +1,18 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include +#include #include #include #include +#include +#include +#include +#include #include #include +#include #include #include #include @@ -13,41 +20,1256 @@ #include #include #include +#include #include +#include +#include #include #include #include #include #include "../filesystems/wrappers.h" -#include "../pidfd/pidfd.h" +#include "coredump_notify_signal.h" -/* Forward declarations to avoid including harness header */ -struct __test_metadata; +#include "coredump_test_helpers.h" -/* Match the fixture definition from coredump_test.h */ -struct _fixture_coredump_data { - char original_core_pattern[256]; - pid_t pid_coredump_server; - int fd_tmpfs_detached; +#if __ELF_NATIVE_CLASS == 64 +#define COREDUMP_ELFCLASS ELFCLASS64 +#else +#define COREDUMP_ELFCLASS ELFCLASS32 +#endif + +#ifndef SO_PEERPIDFD_THREAD +#define SO_PEERPIDFD_THREAD 87 +#endif + +void *do_nothing(void *arg) +{ + (void)arg; + while (1) + pause(); + + return NULL; +} + +void crashing_child(void) +{ + pthread_t thread; + int i; + + for (i = 0; i < NUM_THREAD_SPAWN; ++i) + pthread_create(&thread, NULL, do_nothing, NULL); + + /* crash on purpose */ + i = *(volatile int *)NULL; +} + +void crashing_child_sparse(size_t size) +{ + char *p; + + /* + * Touch the first and the last page. This will cause the whole mapping + * to be dumped because it has been written to. Everything between + * those two pages is a hole though. + */ + p = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (p != MAP_FAILED) { + p[0] = 'x'; + p[size - 1] = 'x'; + } + + /* crash on purpose */ + *(volatile int *)NULL = 0; +} + +/* Select @types through the caller's own /proc/self/coredump_filter. */ +static bool set_coredump_filter(__u64 types) +{ + char buf[32]; + int fd, len; + bool ok; + + fd = open("/proc/self/coredump_filter", O_WRONLY | O_CLOEXEC); + if (fd < 0) + return false; + + len = snprintf(buf, sizeof(buf), "0x%llx", (unsigned long long)types); + ok = write_nointr(fd, buf, len) == len; + close(fd); + return ok; +} + +/* + * Map shared anonymous memory, touch it, tell the server where it is and + * crash. A @task_filter other than FILTER_TASK_INHERIT is selected first. + */ +void crashing_child_memory(__u64 task_filter, int fd_addr) +{ + char *p; + + if (task_filter != FILTER_TASK_INHERIT && !set_coredump_filter(task_filter)) + _exit(EXIT_FAILURE); + + p = mmap(NULL, MEMORY_MAPPING_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + _exit(EXIT_FAILURE); + p[0] = 'x'; + + if (write_nointr(fd_addr, &p, sizeof(p)) != sizeof(p)) + _exit(EXIT_FAILURE); + close(fd_addr); + + /* crash on purpose */ + *(volatile int *)NULL = 0; +} + +/* Sink a reassembled record stream is handed to, record by record. */ +struct coredump_record_sink { + /* @len bytes of coredump data that belong at @offset. */ + int (*data)(void *ctx, const void *buf, size_t len, __u64 offset); + /* @len zero bytes that belong at @offset. */ + int (*zero)(void *ctx, __u64 offset, __u64 len); + void *ctx; }; -#ifndef PAGE_SIZE -#define PAGE_SIZE 4096 -#endif +/* Read @len bytes off the socket and hand them to @sink, if there is one. */ +static ssize_t recv_record_bytes(int fd_coredump, __u64 len, + const struct coredump_record_sink *sink, + __u64 offset) +{ + ssize_t received = 0; + + while (len) { + char buffer[PAGE_SIZE]; + size_t chunk = len < sizeof(buffer) ? len : sizeof(buffer); + ssize_t ret; + + ret = recv(fd_coredump, buffer, chunk, MSG_WAITALL); + if (ret <= 0) { + fprintf(stderr, "%s: short read %zd: %m\n", + __func__, ret); + return -1; + } + + if (sink && sink->data(sink->ctx, buffer, ret, offset + received)) + return -1; + + received += ret; + len -= ret; + } + + return received; +} + +/* Put the data where the records say it goes and leave the holes alone. */ +static int file_sink_data(void *ctx, const void *buf, size_t len, __u64 offset) +{ + int fd = *(int *)ctx; + + if (pwrite(fd, buf, len, offset) != (ssize_t)len) { + fprintf(stderr, "%s: pwrite failed: %m\n", __func__); + return -1; + } + + return 0; +} + +static int file_sink_zero(void *ctx, __u64 offset, __u64 len) +{ + /* Nothing has to be written for a hole. */ + return 0; +} + +/* + * Read a coredump strea and funnel it into @sink. Allow to pass in a + * @fd_peer_pidfd to simulate coredump truncation by killing it after having + * received a coredump record. + */ +static ssize_t __recv_coredump_records(int fd_coredump, + const struct coredump_record_sink *sink, + off_t *coredump_size, bool *truncated, + int fd_peer_pidfd) +{ + ssize_t received = 0; + off_t size = 0; + bool is_truncated = false; + bool ended = false; + char trailing; + + while (!ended) { + struct coredump_record_header record = {}; + size_t known_size; + ssize_t ret; + + /* Peek the header size the way read_coredump_req() does. */ + ret = recv(fd_coredump, &record, sizeof(record.size), + MSG_PEEK | MSG_WAITALL); + if (ret == 0) { + /* Nothing closed the stream, so the coredump was cut short. */ + if (truncated) { + is_truncated = true; + break; + } + fprintf(stderr, "%s: stream ended without an end record\n", + __func__); + return -1; + } + if (ret != sizeof(record.size)) { + fprintf(stderr, "%s: short record peek %zd: %m\n", + __func__, ret); + return -1; + } + + if (record.size < COREDUMP_RECORD_HEADER_SIZE_VER0) { + fprintf(stderr, "%s: header size %u below minimum %u\n", + __func__, record.size, + COREDUMP_RECORD_HEADER_SIZE_VER0); + return -1; + } + + /* Consume as much of the header as we know about. */ + known_size = record.size < sizeof(record) ? record.size : sizeof(record); + ret = recv(fd_coredump, &record, known_size, MSG_WAITALL); + if (ret != (ssize_t)known_size) { + fprintf(stderr, "%s: short record read %zd: %m\n", + __func__, ret); + return -1; + } + received += ret; + + /* + * A flag changes what the record means, so refuse one we + * don't know rather than guess. + */ + if (record.flags) { + fprintf(stderr, "%s: unknown header flags 0x%llx\n", + __func__, (unsigned long long)record.flags); + return -1; + } + + /* Discard any part of the header we have no use for. */ + ret = recv_record_bytes(fd_coredump, record.size - known_size, + NULL, 0); + if (ret < 0) + return -1; + received += ret; + + /* Records are sent in order and they don't leave gaps. */ + if (record.offset != (__u64)size) { + fprintf(stderr, "%s: record at %llu, expected %llu\n", + __func__, (unsigned long long)record.offset, + (unsigned long long)size); + return -1; + } + + switch (record.type) { + case COREDUMP_RECORD_ZERO: + /* A hole. It comes with no data and needs none. */ + if (sink->zero(sink->ctx, record.offset, record.len)) + return -1; + break; + case COREDUMP_RECORD_DATA: + ret = recv_record_bytes(fd_coredump, record.len, sink, + record.offset); + if (ret < 0) + return -1; + received += ret; + if (fd_peer_pidfd >= 0) { + if (sys_pidfd_send_signal(fd_peer_pidfd, SIGKILL, + NULL, 0)) { + fprintf(stderr, "%s: kill failed: %m\n", + __func__); + return -1; + } + fd_peer_pidfd = -1; + } + break; + case COREDUMP_RECORD_END: + /* The coredump ends here and nothing follows it. */ + if (record.len) { + fprintf(stderr, "%s: end record covers %llu bytes\n", + __func__, + (unsigned long long)record.len); + return -1; + } + ended = true; + break; + default: + fprintf(stderr, "%s: unknown record type %u\n", + __func__, record.type); + return -1; + } + + size += record.len; + } + + /* The end record is the last thing on the wire. */ + if (recv(fd_coredump, &trailing, sizeof(trailing), MSG_DONTWAIT) > 0) { + fprintf(stderr, "%s: data after the end record\n", __func__); + return -1; + } + + if (truncated) + *truncated = is_truncated; + + *coredump_size = size; + + fprintf(stderr, "Received %zd bytes for a %s coredump of %llu bytes\n", + received, is_truncated ? "truncated" : "complete", + (unsigned long long)size); + return received; +} + +/* Reassemble a record stream into the coredump it describes. */ +ssize_t recv_coredump_records(int fd_coredump, int fd_core_file, + off_t *coredump_size, bool *truncated, + int fd_peer_pidfd) +{ + struct coredump_record_sink sink = { + .data = file_sink_data, + .zero = file_sink_zero, + .ctx = &fd_core_file, + }; + ssize_t received; + off_t size = 0; + + received = __recv_coredump_records(fd_coredump, &sink, &size, truncated, + fd_peer_pidfd); + if (received < 0) + return -1; + + /* + * Nothing is written for a hole, so grow the file to the size the + * records describe in case the coredump ended in one. + */ + if (ftruncate(fd_core_file, size) < 0) { + fprintf(stderr, "%s: ftruncate to %llu failed: %m\n", + __func__, (unsigned long long)size); + return -1; + } + + if (coredump_size) + *coredump_size = size; + + return received; +} + +/* The ELF header of a native core file. */ +static bool is_core_ehdr(const ElfW(Ehdr) *ehdr) +{ + return !memcmp(ehdr->e_ident, ELFMAG, SELFMAG) && + ehdr->e_ident[EI_CLASS] == COREDUMP_ELFCLASS && + ehdr->e_type == ET_CORE; +} + +/* Whatever the server ends up with has to be an ELF core file. */ +bool is_elf_core(int fd) +{ + ElfW(Ehdr) ehdr; + + if (pread(fd, &ehdr, sizeof(ehdr), 0) != sizeof(ehdr)) { + fprintf(stderr, "%s: short read: %m\n", __func__); + return false; + } + + if (!is_core_ehdr(&ehdr)) { + fprintf(stderr, "%s: not an ELF core file\n", __func__); + return false; + } + + return true; +} + +/* + * A coredump server that uploads to a blob store can't upload a sparse + * file and can't seek in the object it is uploading. It streams the data + * records into the object as they arrive, remembers the holes it left + * out, and uploads the program header table that describes the result + * last. What comes out is an ordinary ELF core file without the holes. + */ + +/* A run of the coredump the object doesn't carry. */ +struct compact_hole { + __u64 offset; + __u64 len; +}; + +/* A program header of the object and where its bytes sat in the coredump. */ +struct compact_piece { + ElfW(Phdr) phdr; + __u64 src; +}; + +struct compact_ctx { + int fd_body; /* the object's payload, append only */ + int fd_reference; /* the coredump with its holes, for the test */ + unsigned char *head; /* everything ahead of the segment data */ + size_t head_len; + size_t head_cap; + __u64 data_offset; /* where the segment data starts, 0 while unknown */ + struct compact_hole *holes; + size_t nr_holes; + size_t holes_cap; + __u64 body_len; +}; + +/* Write @len bytes out, short writes and all. */ +static int compact_write(int fd, const void *buf, size_t len) +{ + const unsigned char *pos = buf; + + while (len) { + ssize_t ret = write(fd, pos, len); + + if (ret <= 0) { + fprintf(stderr, "%s: write failed: %m\n", __func__); + return -1; + } + + pos += ret; + len -= ret; + } + + return 0; +} + +/* Keep @len bytes of the head, or @len zeroes if @buf is NULL. */ +static int compact_head_append(struct compact_ctx *ctx, const void *buf, + size_t len) +{ + if (ctx->head_len + len > ctx->head_cap) { + size_t cap = ctx->head_cap ? ctx->head_cap : PAGE_SIZE; + unsigned char *head; + + while (cap < ctx->head_len + len) + cap *= 2; + + head = realloc(ctx->head, cap); + if (!head) { + fprintf(stderr, "%s: out of memory\n", __func__); + return -1; + } + ctx->head = head; + ctx->head_cap = cap; + } + + if (buf) + memcpy(ctx->head + ctx->head_len, buf, len); + else + memset(ctx->head + ctx->head_len, 0, len); + ctx->head_len += len; + + return 0; +} + +/* Remember a hole so the program header table can account for it later. */ +static int compact_keep_hole(struct compact_ctx *ctx, __u64 offset, __u64 len) +{ + if (ctx->nr_holes == ctx->holes_cap) { + size_t cap = ctx->holes_cap ? ctx->holes_cap * 2 : 64; + struct compact_hole *holes; + + holes = realloc(ctx->holes, cap * sizeof(*holes)); + if (!holes) { + fprintf(stderr, "%s: out of memory\n", __func__); + return -1; + } + ctx->holes = holes; + ctx->holes_cap = cap; + } + + ctx->holes[ctx->nr_holes].offset = offset; + ctx->holes[ctx->nr_holes].len = len; + ctx->nr_holes++; + + return 0; +} + +/* The segment data starts where the first PT_LOAD points. */ +static int compact_probe(struct compact_ctx *ctx) +{ + const ElfW(Ehdr) *ehdr = (const ElfW(Ehdr) *)ctx->head; + const ElfW(Phdr) *phdr; + size_t i; + + if (ctx->data_offset || ctx->head_len < sizeof(*ehdr)) + return 0; + + if (!is_core_ehdr(ehdr)) { + fprintf(stderr, "%s: not an ELF core file\n", __func__); + return -1; + } + + if (ehdr->e_phoff != sizeof(*ehdr) || + ehdr->e_phentsize != sizeof(ElfW(Phdr)) || + ehdr->e_phnum == 0 || ehdr->e_phnum == PN_XNUM) { + fprintf(stderr, "%s: unhandled program header table\n", __func__); + return -1; + } + + if (ctx->head_len < ehdr->e_phoff + + (size_t)ehdr->e_phnum * ehdr->e_phentsize) + return 0; + + phdr = (const ElfW(Phdr) *)(ctx->head + ehdr->e_phoff); + for (i = 0; i < ehdr->e_phnum; i++) { + if (phdr[i].p_type != PT_LOAD) + continue; + if (!ctx->data_offset || phdr[i].p_offset < ctx->data_offset) + ctx->data_offset = phdr[i].p_offset; + } + + if (!ctx->data_offset) { + fprintf(stderr, "%s: coredump without a single segment\n", + __func__); + return -1; + } + + return 0; +} + +/* + * Take whatever of [@offset, @offset + @len) still belongs to the head. + * @buf is NULL for a hole. Returns how much was taken. + */ +static ssize_t compact_head_take(struct compact_ctx *ctx, const void *buf, + __u64 offset, __u64 len) +{ + __u64 chunk; + + if (!len || (ctx->data_offset && offset >= ctx->data_offset)) + return 0; + + chunk = len; + if (ctx->data_offset && offset + chunk > ctx->data_offset) + chunk = ctx->data_offset - offset; + + if (offset != ctx->head_len) { + fprintf(stderr, "%s: head has a gap at %llu\n", __func__, + (unsigned long long)offset); + return -1; + } + + if (compact_head_append(ctx, buf, chunk)) + return -1; + + return chunk; +} + +static int compact_data(void *arg, const void *buf, size_t len, __u64 offset) +{ + struct compact_ctx *ctx = arg; + const unsigned char *pos = buf; + ssize_t head; + + /* Only the test needs a coredump with the holes still in it. */ + if (pwrite(ctx->fd_reference, pos, len, offset) != (ssize_t)len) { + fprintf(stderr, "%s: pwrite failed: %m\n", __func__); + return -1; + } + + /* The head has to be rewritten at the end, so hold on to it. */ + head = compact_head_take(ctx, pos, offset, len); + if (head < 0) + return -1; + if (head && compact_probe(ctx)) + return -1; + + pos += head; + len -= head; + if (!len) + return 0; + + /* Everything else goes into the object as it arrives. */ + if (compact_write(ctx->fd_body, pos, len)) + return -1; + ctx->body_len += len; + + return 0; +} + +static int compact_zero(void *arg, __u64 offset, __u64 len) +{ + struct compact_ctx *ctx = arg; + ssize_t head; + + /* A hole in the head is alignment padding. Write it out. */ + head = compact_head_take(ctx, NULL, offset, len); + if (head < 0) + return -1; + + offset += head; + len -= head; + if (!len) + return 0; + + /* This is what the object doesn't have to carry. */ + return compact_keep_hole(ctx, offset, len); +} + +/* Where @offset ends up in the object once the holes ahead of it are gone. */ +static __u64 compact_offset(const struct compact_ctx *ctx, __u64 body_start, + __u64 offset) +{ + __u64 elided = 0; + size_t i; + + for (i = 0; i < ctx->nr_holes; i++) { + __u64 len = ctx->holes[i].len; + + if (ctx->holes[i].offset >= offset) + break; + if (ctx->holes[i].offset + len > offset) + len = offset - ctx->holes[i].offset; + elided += len; + } + + return body_start + (offset - ctx->data_offset) - elided; +} + +/* A run of segment data that made it into the object. */ +static void compact_add_data(struct compact_piece *pieces, size_t *nr, + const ElfW(Phdr) *phdr, __u64 start, __u64 end) +{ + struct compact_piece *piece = &pieces[(*nr)++]; + + piece->phdr = *phdr; + piece->phdr.p_vaddr = phdr->p_vaddr + (start - phdr->p_offset); + piece->phdr.p_paddr = 0; + piece->phdr.p_filesz = end - start; + piece->phdr.p_memsz = end - start; + piece->src = start; +} + +/* + * A run of @len bytes the object doesn't carry. It grows the piece in + * front of it if this segment already has one, because everything a + * segment covers past p_filesz is zeroes anyway. + */ +static void compact_add_zero(struct compact_piece *pieces, size_t *nr, + size_t first, const ElfW(Phdr) *phdr, __u64 vaddr, + __u64 len) +{ + struct compact_piece *piece; + + if (*nr > first) { + pieces[*nr - 1].phdr.p_memsz += len; + return; + } + + piece = &pieces[(*nr)++]; + piece->phdr = *phdr; + piece->phdr.p_vaddr = vaddr; + piece->phdr.p_paddr = 0; + piece->phdr.p_filesz = 0; + piece->phdr.p_memsz = len; + piece->src = 0; +} + +/* Split the segments at the holes and write out what the object became. */ +static int compact_build(struct compact_ctx *ctx, int fd_object) +{ + __u64 note_offset = 0, note_len = 0, note_new; + __u64 align = 0, head_len, body_start, pos; + size_t nr_old, nr_new = 0, note_piece = 0, i; + struct compact_piece *pieces; + char buffer[PAGE_SIZE]; + const ElfW(Phdr) *old; + ElfW(Ehdr) ehdr; + int ret = -1; + + if (!ctx->data_offset) { + fprintf(stderr, "%s: coredump without segment data\n", __func__); + return -1; + } + + memcpy(&ehdr, ctx->head, sizeof(ehdr)); + if (ehdr.e_shoff) { + fprintf(stderr, "%s: section headers are not handled\n", + __func__); + return -1; + } + + old = (const ElfW(Phdr) *)(ctx->head + ehdr.e_phoff); + nr_old = ehdr.e_phnum; + + pieces = calloc(nr_old + 2 * ctx->nr_holes + 1, sizeof(*pieces)); + if (!pieces) { + fprintf(stderr, "%s: out of memory\n", __func__); + return -1; + } + + for (i = 0; i < nr_old; i++) { + ElfW(Phdr) phdr = old[i]; + __u64 end = phdr.p_offset + phdr.p_filesz; + __u64 cur = phdr.p_offset; + size_t first = nr_new, h; + + /* The notes move because the table in front of them grows. */ + if (phdr.p_type == PT_NOTE) { + if (note_len) { + fprintf(stderr, "%s: more than one note segment\n", + __func__); + goto out; + } + note_offset = phdr.p_offset; + note_len = phdr.p_filesz; + note_piece = nr_new; + pieces[nr_new].phdr = phdr; + pieces[nr_new++].src = 0; + continue; + } + + if (phdr.p_type != PT_LOAD) { + if (phdr.p_filesz && phdr.p_offset < ctx->data_offset) { + fprintf(stderr, "%s: segment %zu is in the head\n", + __func__, i); + goto out; + } + pieces[nr_new].phdr = phdr; + pieces[nr_new++].src = phdr.p_offset; + continue; + } + + if (!align) + align = phdr.p_align; + + for (h = 0; h < ctx->nr_holes && cur < end; h++) { + __u64 start = ctx->holes[h].offset; + __u64 stop = start + ctx->holes[h].len; + + if (stop <= cur) + continue; + if (start >= end) + break; + + /* A hole can span more than this one segment. */ + if (start < cur) + start = cur; + if (stop > end) + stop = end; + + if (start > cur) { + compact_add_data(pieces, &nr_new, &phdr, cur, + start); + cur = start; + } + compact_add_zero(pieces, &nr_new, first, &phdr, + phdr.p_vaddr + (cur - phdr.p_offset), + stop - cur); + cur = stop; + } + + if (cur < end) + compact_add_data(pieces, &nr_new, &phdr, cur, end); + + /* Whatever the kernel didn't dump of this mapping. */ + if (phdr.p_memsz > phdr.p_filesz) + compact_add_zero(pieces, &nr_new, first, &phdr, + phdr.p_vaddr + phdr.p_filesz, + phdr.p_memsz - phdr.p_filesz); + } + + if (!note_len || note_offset + note_len > ctx->head_len) { + fprintf(stderr, "%s: notes aren't where they should be\n", + __func__); + goto out; + } + + if (nr_new >= PN_XNUM) { + fprintf(stderr, "%s: %zu program headers don't fit\n", __func__, + nr_new); + goto out; + } + + if (!align || (align & (align - 1))) + align = sysconf(_SC_PAGESIZE); + + note_new = sizeof(ehdr) + (__u64)nr_new * sizeof(ElfW(Phdr)); + head_len = note_new + note_len; + body_start = (head_len + align - 1) & ~(align - 1); + + for (i = 0; i < nr_new; i++) { + struct compact_piece *piece = &pieces[i]; + + if (i == note_piece) + piece->phdr.p_offset = note_new; + else if (piece->phdr.p_filesz) + piece->phdr.p_offset = compact_offset(ctx, body_start, + piece->src); + else + piece->phdr.p_offset = 0; + } + + /* Only now is the head known. That's why it is uploaded last. */ + ehdr.e_phnum = nr_new; + if (compact_write(fd_object, &ehdr, sizeof(ehdr))) + goto out; + + for (i = 0; i < nr_new; i++) + if (compact_write(fd_object, &pieces[i].phdr, + sizeof(pieces[i].phdr))) + goto out; + + if (compact_write(fd_object, ctx->head + note_offset, note_len)) + goto out; + + /* Keep the segments aligned the way a debugger expects them. */ + memset(buffer, 0, sizeof(buffer)); + for (pos = head_len; pos < body_start; ) { + __u64 chunk = body_start - pos; + + if (chunk > sizeof(buffer)) + chunk = sizeof(buffer); + if (compact_write(fd_object, buffer, chunk)) + goto out; + pos += chunk; + } + + /* Putting the parts together is the blob store's job. Do it here. */ + for (pos = 0; pos < ctx->body_len; ) { + ssize_t chunk = pread(ctx->fd_body, buffer, sizeof(buffer), pos); + + if (chunk <= 0) { + fprintf(stderr, "%s: short read %zd: %m\n", __func__, + chunk); + goto out; + } + if (compact_write(fd_object, buffer, chunk)) + goto out; + pos += chunk; + } + + fprintf(stderr, "Object is %llu bytes in %zu program headers, %zu holes left out\n", + (unsigned long long)(body_start + ctx->body_len), nr_new, + ctx->nr_holes); + ret = 0; +out: + free(pieces); + return ret; +} + +/* + * Reassemble a record stream into an ELF core file that has no holes in + * it, the way a coredump server that uploads to a blob store has to. If + * @fd_reference is valid it gets the coredump the records describe, + * holes and all, so the test can compare the two. + */ +ssize_t recv_coredump_compact(int fd_coredump, int fd_object, int fd_reference, + off_t *coredump_size) +{ + struct compact_ctx ctx = { + .fd_body = -1, + .fd_reference = fd_reference, + }; + struct coredump_record_sink sink = { + .data = compact_data, + .zero = compact_zero, + .ctx = &ctx, + }; + ssize_t received; + off_t size = 0; + FILE *body; + + body = tmpfile(); + if (!body) { + fprintf(stderr, "%s: tmpfile failed: %m\n", __func__); + return -1; + } + ctx.fd_body = fileno(body); + + /* An upload is appended to. Make sure nothing here can seek. */ + if (fcntl(ctx.fd_body, F_SETFL, O_APPEND)) { + fprintf(stderr, "%s: F_SETFL failed: %m\n", __func__); + received = -1; + goto out; + } + + received = __recv_coredump_records(fd_coredump, &sink, &size, NULL, -1); + if (received < 0) + goto out; + + /* + * Nothing is written for a hole, so grow the reference to the size + * the records describe in case the coredump ended in one. + */ + if (ftruncate(fd_reference, size) < 0) { + fprintf(stderr, "%s: ftruncate to %llu failed: %m\n", + __func__, (unsigned long long)size); + received = -1; + goto out; + } + + if (compact_build(&ctx, fd_object)) { + received = -1; + goto out; + } + + if (coredump_size) + *coredump_size = size; +out: + fclose(body); + free(ctx.head); + free(ctx.holes); + return received; +} + +/* Read the ELF header and the program header table of @fd. */ +static ElfW(Phdr) *read_phdrs(int fd, size_t *nr) +{ + ElfW(Ehdr) ehdr; + ElfW(Phdr) *phdr; + size_t size; + + if (pread(fd, &ehdr, sizeof(ehdr), 0) != sizeof(ehdr)) { + fprintf(stderr, "%s: no ELF header: %m\n", __func__); + return NULL; + } + + if (!is_core_ehdr(&ehdr) || !ehdr.e_phnum || + ehdr.e_phentsize != sizeof(*phdr)) { + fprintf(stderr, "%s: not an ELF core file\n", __func__); + return NULL; + } + + size = (size_t)ehdr.e_phnum * ehdr.e_phentsize; + phdr = malloc(size); + if (!phdr) { + fprintf(stderr, "%s: out of memory\n", __func__); + return NULL; + } + + if (pread(fd, phdr, size, ehdr.e_phoff) != (ssize_t)size) { + fprintf(stderr, "%s: short program header table: %m\n", __func__); + free(phdr); + return NULL; + } + + *nr = ehdr.e_phnum; + return phdr; +} + +/* The segment @vaddr falls into. */ +static const ElfW(Phdr) *find_segment(const ElfW(Phdr) *phdr, size_t nr, + __u64 vaddr) +{ + size_t i; + + for (i = 0; i < nr; i++) { + if (phdr[i].p_type != PT_LOAD) + continue; + if (vaddr >= phdr[i].p_vaddr && + vaddr < phdr[i].p_vaddr + phdr[i].p_memsz) + return &phdr[i]; + } + + return NULL; +} + +/* The PT_LOAD segment @vaddr falls into. */ +bool find_coredump_segment(int fd, __u64 vaddr, ElfW(Phdr) *segment) +{ + const ElfW(Phdr) *found; + ElfW(Phdr) *phdr; + size_t nr; + + phdr = read_phdrs(fd, &nr); + if (!phdr) + return false; + + found = find_segment(phdr, nr, vaddr); + if (found) + *segment = *found; + else + fprintf(stderr, "%s: no segment for 0x%llx\n", __func__, + (unsigned long long)vaddr); + + free(phdr); + return found; +} + +/* How many bytes the PT_LOAD and the PT_NOTE segments of @fd carry. */ +bool sum_coredump_segments(int fd, __u64 *data, __u64 *notes) +{ + ElfW(Phdr) *phdr; + size_t nr, i; + + phdr = read_phdrs(fd, &nr); + if (!phdr) + return false; + + *data = 0; + *notes = 0; + for (i = 0; i < nr; i++) { + if (phdr[i].p_type == PT_LOAD) + *data += phdr[i].p_filesz; + else if (phdr[i].p_type == PT_NOTE) + *notes += phdr[i].p_filesz; + } + + free(phdr); + return true; +} + +/* The coredump in @fd is at least as long as every segment it declares. */ +bool check_coredump_extent(int fd) +{ + ElfW(Phdr) *phdr; + struct stat st; + size_t nr, i; + bool ok = true; + + if (fstat(fd, &st)) { + fprintf(stderr, "%s: fstat: %m\n", __func__); + return false; + } + + phdr = read_phdrs(fd, &nr); + if (!phdr) + return false; + + for (i = 0; i < nr; i++) { + if (phdr[i].p_offset + phdr[i].p_filesz <= (__u64)st.st_size) + continue; + fprintf(stderr, "%s: segment %zu ends at %llu, the coredump at %llu\n", + __func__, i, + (unsigned long long)(phdr[i].p_offset + phdr[i].p_filesz), + (unsigned long long)st.st_size); + ok = false; + } + + free(phdr); + return ok; +} + +/* The next stretch of memory the segments cover, split ones merged back. */ +static bool next_range(const ElfW(Phdr) *phdr, size_t nr, size_t *i, + __u64 *start, __u64 *end) +{ + while (*i < nr && phdr[*i].p_type != PT_LOAD) + (*i)++; + + if (*i >= nr) + return false; + + *start = phdr[*i].p_vaddr; + *end = phdr[*i].p_vaddr + phdr[*i].p_memsz; + (*i)++; + + while (*i < nr) { + if (phdr[*i].p_type != PT_LOAD) { + (*i)++; + continue; + } + if (phdr[*i].p_vaddr != *end) + break; + *end = phdr[*i].p_vaddr + phdr[*i].p_memsz; + (*i)++; + } + + return true; +} + +/* Compare @len bytes at @offset against @len bytes at @offset_ref. */ +static int compare_range(int fd, __u64 offset, int fd_ref, __u64 offset_ref, + __u64 len) +{ + char buffer[PAGE_SIZE], buffer_ref[PAGE_SIZE]; + + while (len) { + size_t chunk = len < sizeof(buffer) ? len : sizeof(buffer); + + if (pread(fd, buffer, chunk, offset) != (ssize_t)chunk || + pread(fd_ref, buffer_ref, chunk, offset_ref) != (ssize_t)chunk) { + fprintf(stderr, "%s: short read at %llu: %m\n", + __func__, (unsigned long long)offset); + return -1; + } + + if (memcmp(buffer, buffer_ref, chunk)) { + fprintf(stderr, "%s: %llu differs from %llu\n", __func__, + (unsigned long long)offset, + (unsigned long long)offset_ref); + return -1; + } + + offset += chunk; + offset_ref += chunk; + len -= chunk; + } + + return 0; +} + +/* The @len bytes at @offset the object left out have to have been zeroes. */ +static int check_zero_range(int fd, __u64 offset, __u64 len) +{ + static const char zeroes[PAGE_SIZE]; + char buffer[PAGE_SIZE]; -#define NUM_THREAD_SPAWN 128 + while (len) { + size_t chunk = len < sizeof(buffer) ? len : sizeof(buffer); -void *do_nothing(void *arg) + if (pread(fd, buffer, chunk, offset) != (ssize_t)chunk) { + fprintf(stderr, "%s: short read at %llu: %m\n", + __func__, (unsigned long long)offset); + return -1; + } + + if (memcmp(buffer, zeroes, chunk)) { + fprintf(stderr, "%s: %llu isn't a hole\n", __func__, + (unsigned long long)offset); + return -1; + } + + offset += chunk; + len -= chunk; + } + + return 0; +} + +/* + * The object has to describe the same memory as the coredump it was built + * from, and it has to describe it correctly. + */ +int check_compact_coredump(int fd_object, int fd_reference) +{ + ElfW(Phdr) *object = NULL, *reference = NULL; + size_t nr_object, nr_reference, i; + size_t io = 0, ir = 0; + int ret = -1; + + object = read_phdrs(fd_object, &nr_object); + reference = read_phdrs(fd_reference, &nr_reference); + if (!object || !reference) + goto out; + + /* Nothing may have been dropped and nothing may have been added. */ + for (;;) { + __u64 start = 0, end = 0, start_ref = 0, end_ref = 0; + bool has, has_ref; + + has = next_range(object, nr_object, &io, &start, &end); + has_ref = next_range(reference, nr_reference, &ir, &start_ref, + &end_ref); + if (!has && !has_ref) + break; + + if (has != has_ref || start != start_ref || end != end_ref) { + fprintf(stderr, "%s: object covers 0x%llx-0x%llx, coredump 0x%llx-0x%llx\n", + __func__, (unsigned long long)start, + (unsigned long long)end, + (unsigned long long)start_ref, + (unsigned long long)end_ref); + goto out; + } + } + + for (i = 0; i < nr_object; i++) { + const ElfW(Phdr) *segment; + __u64 offset, dumped; + + if (object[i].p_type != PT_LOAD || !object[i].p_memsz) + continue; + + segment = find_segment(reference, nr_reference, + object[i].p_vaddr); + if (!segment) { + fprintf(stderr, "%s: 0x%llx isn't in the coredump\n", + __func__, + (unsigned long long)object[i].p_vaddr); + goto out; + } + + offset = object[i].p_vaddr - segment->p_vaddr; + dumped = offset < segment->p_filesz ? + segment->p_filesz - offset : 0; + + /* What the object carries is what the coredump had. */ + if (object[i].p_filesz > dumped) { + fprintf(stderr, "%s: object carries %llu bytes the coredump doesn't have\n", + __func__, + (unsigned long long)(object[i].p_filesz - dumped)); + goto out; + } + + if (compare_range(fd_object, object[i].p_offset, fd_reference, + segment->p_offset + offset, + object[i].p_filesz)) + goto out; + + /* And what it left out was a hole. */ + if (object[i].p_memsz > object[i].p_filesz && + dumped > object[i].p_filesz) { + __u64 left_out = dumped - object[i].p_filesz; + + if (left_out > object[i].p_memsz - object[i].p_filesz) + left_out = object[i].p_memsz - object[i].p_filesz; + + if (check_zero_range(fd_reference, + segment->p_offset + offset + + object[i].p_filesz, left_out)) + goto out; + } + } + + ret = 0; +out: + free(object); + free(reference); + return ret; +} + +/* Read a plain coredump byte stream to end-of-file. */ +ssize_t recv_coredump_bytes(int fd_coredump, int fd_core_file) +{ + ssize_t received = 0; + + for (;;) { + char buffer[PAGE_SIZE]; + ssize_t ret = read_nointr(fd_coredump, buffer, sizeof(buffer)); + + if (ret < 0) { + fprintf(stderr, "%s: read failed: %m\n", __func__); + return -1; + } + if (ret == 0) + break; + + if (write_nointr(fd_core_file, buffer, ret) != ret) { + fprintf(stderr, "%s: write failed: %m\n", __func__); + return -1; + } + received += ret; + } + + fprintf(stderr, "Received %zd bytes of coredump\n", received); + return received; +} + +static void *crashing_thread(void *arg) { + int *p; + (void)arg; - while (1) - pause(); + + /* crash on purpose with SEGV_MAPERR */ + p = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + return NULL; + munmap(p, PAGE_SIZE); + *p = 0; return NULL; } -void crashing_child(void) +void crashing_child_thread(void) { pthread_t thread; int i; @@ -55,8 +1277,9 @@ void crashing_child(void) for (i = 0; i < NUM_THREAD_SPAWN; ++i) pthread_create(&thread, NULL, do_nothing, NULL); - /* crash on purpose */ - i = *(volatile int *)NULL; + /* crash from a non-leader thread */ + pthread_create(&thread, NULL, crashing_thread, NULL); + pause(); } int create_detached_tmpfs(void) @@ -101,6 +1324,7 @@ int create_and_listen_unix_socket(const char *path) return fd; out: + fprintf(stderr, "%s: %s: %m\n", __func__, path); if (fd >= 0) close(fd); return -1; @@ -138,6 +1362,20 @@ int get_peer_pidfd(int fd) return fd_peer_pidfd; } +int get_peer_pidfd_thread(int fd) +{ + int fd_peer_pidfd; + socklen_t fd_peer_pidfd_len = sizeof(fd_peer_pidfd); + int ret = getsockopt(fd, SOL_SOCKET, SO_PEERPIDFD_THREAD, &fd_peer_pidfd, + &fd_peer_pidfd_len); + if (ret < 0) { + fprintf(stderr, "%s: getsockopt(SO_PEERPIDFD_THREAD) failed: %m\n", __func__); + return -1; + } + fprintf(stderr, "%s: successfully retrieved pidfd %d\n", __func__, fd_peer_pidfd); + return fd_peer_pidfd; +} + bool get_pidfd_info(int fd_peer_pidfd, struct pidfd_info *info) { int ret; @@ -153,8 +1391,95 @@ bool get_pidfd_info(int fd_peer_pidfd, struct pidfd_info *info) return true; } +/* + * How much the peer has mapped. The task is parked in the coredump + * handshake, so its mm is still there to be looked at. + */ +ssize_t peer_vm_size(int fd_peer_pidfd) +{ + struct pidfd_info info = {}; + unsigned long pages; + char path[64]; + FILE *f; + + if (!get_pidfd_info(fd_peer_pidfd, &info)) + return -1; + + snprintf(path, sizeof(path), "/proc/%d/statm", info.pid); + f = fopen(path, "r"); + if (!f) { + fprintf(stderr, "%s: %s: %m\n", __func__, path); + return -1; + } + + if (fscanf(f, "%lu", &pages) != 1) { + fprintf(stderr, "%s: %s: no size\n", __func__, path); + fclose(f); + return -1; + } + fclose(f); + + return (ssize_t)pages * sysconf(_SC_PAGESIZE); +} + /* Protocol helper functions */ +/* The peer's /proc//coredump_filter, which is in memory types. */ +bool peer_coredump_filter(int fd_peer_pidfd, __u64 *memory_types) +{ + struct pidfd_info info = {}; + unsigned long value; + char path[64]; + FILE *f; + int ret; + + if (!get_pidfd_info(fd_peer_pidfd, &info)) + return false; + + snprintf(path, sizeof(path), "/proc/%d/coredump_filter", info.pid); + f = fopen(path, "r"); + if (!f) { + fprintf(stderr, "%s: %s: %m\n", __func__, path); + return false; + } + + ret = fscanf(f, "%lx", &value); + fclose(f); + if (ret != 1) { + fprintf(stderr, "%s: %s: no value\n", __func__, path); + return false; + } + + *memory_types = value; + return true; +} + +/* Read @len bytes at @addr from the peer's /proc//mem. */ +ssize_t peer_read_mem(int fd_peer_pidfd, __u64 addr, void *buf, size_t len) +{ + struct pidfd_info info = {}; + char path[64]; + ssize_t ret; + int fd; + + if (!get_pidfd_info(fd_peer_pidfd, &info)) + return -1; + + snprintf(path, sizeof(path), "/proc/%d/mem", info.pid); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) { + fprintf(stderr, "%s: %s: %m\n", __func__, path); + return -1; + } + + ret = pread(fd, buf, len, addr); + if (ret < 0) + fprintf(stderr, "%s: %s at 0x%llx: %m\n", __func__, path, + (unsigned long long)addr); + close(fd); + return ret; +} + ssize_t recv_marker(int fd) { enum coredump_mark mark = COREDUMP_MARK_REQACK; @@ -197,10 +1522,34 @@ bool read_marker(int fd, enum coredump_mark mark) return ret == mark; } -bool read_coredump_req(int fd, struct coredump_req *req) +/* + * The kernel hung up without sending anything more: end of stream, or a + * reset if it refused the ack on its peeked size and never read it. + */ +bool read_hangup(int fd) +{ + ssize_t ret; + char c; + + ret = recv(fd, &c, sizeof(c), MSG_WAITALL); + if (ret == 0) { + fprintf(stderr, "Kernel closed the connection\n"); + return true; + } + if (ret < 0 && errno == ECONNRESET) { + fprintf(stderr, "Kernel closed the connection with the ack unread\n"); + return true; + } + + fprintf(stderr, "%s: expected a hangup, got %zd: %m\n", __func__, ret); + return false; +} + +/* Read the request as a server built with a @user_size byte struct does. */ +bool read_coredump_req_sized(int fd, struct coredump_req *req, size_t user_size) { ssize_t ret; - size_t field_size, user_size, ack_size, kernel_size, remaining_size; + size_t field_size, known_size, kernel_size, remaining_size; memset(req, 0, sizeof(*req)); field_size = sizeof(req->size); @@ -208,37 +1557,36 @@ bool read_coredump_req(int fd, struct coredump_req *req) /* Peek the size of the coredump request. */ ret = recv(fd, req, field_size, MSG_PEEK | MSG_WAITALL); if (ret != field_size) { - fprintf(stderr, "read_coredump_req: peek failed (got %zd, expected %zu): %m\n", + fprintf(stderr, "%s: peek failed (got %zd, expected %zu): %m\n", __func__, ret, field_size); return false; } kernel_size = req->size; - if (kernel_size < COREDUMP_ACK_SIZE_VER0) { - fprintf(stderr, "read_coredump_req: kernel_size %zu < min %d\n", - kernel_size, COREDUMP_ACK_SIZE_VER0); + if (kernel_size < COREDUMP_REQ_SIZE_VER0) { + fprintf(stderr, "%s: kernel_size %zu < min %d\n", __func__, + kernel_size, COREDUMP_REQ_SIZE_VER0); return false; } if (kernel_size >= PAGE_SIZE) { - fprintf(stderr, "read_coredump_req: kernel_size %zu >= PAGE_SIZE %d\n", + fprintf(stderr, "%s: kernel_size %zu >= PAGE_SIZE %d\n", __func__, kernel_size, PAGE_SIZE); return false; } - /* Use the minimum of user and kernel size to read the full request. */ - user_size = sizeof(struct coredump_req); - ack_size = user_size < kernel_size ? user_size : kernel_size; - ret = recv(fd, req, ack_size, MSG_WAITALL); - if (ret != ack_size) + /* Consume as much of the request as we know about. */ + known_size = user_size < kernel_size ? user_size : kernel_size; + ret = recv(fd, req, known_size, MSG_WAITALL); + if (ret != known_size) return false; fprintf(stderr, "Read coredump request with size %u and mask 0x%llx\n", req->size, (unsigned long long)req->mask); - if (user_size > kernel_size) - remaining_size = user_size - kernel_size; - else + if (kernel_size > user_size) remaining_size = kernel_size - user_size; + else + remaining_size = 0; if (PAGE_SIZE <= remaining_size) return false; @@ -250,7 +1598,7 @@ bool read_coredump_req(int fd, struct coredump_req *req) if (remaining_size) { char buffer[PAGE_SIZE]; - ret = recv(fd, buffer, sizeof(buffer), MSG_WAITALL); + ret = recv(fd, buffer, remaining_size, MSG_WAITALL); if (ret != remaining_size) return false; fprintf(stderr, "Discarded %zu bytes of data after coredump request\n", remaining_size); @@ -259,8 +1607,13 @@ bool read_coredump_req(int fd, struct coredump_req *req) return true; } -bool send_coredump_ack(int fd, const struct coredump_req *req, - __u64 mask, size_t size_ack) +bool read_coredump_req(int fd, struct coredump_req *req) +{ + return read_coredump_req_sized(fd, req, sizeof(*req)); +} + +/* Send @len bytes of @ack as they are, more than the struct if asked to. */ +bool send_coredump_ack_bytes(int fd, const struct coredump_ack *ack, size_t len) { ssize_t ret; /* @@ -272,30 +1625,74 @@ bool send_coredump_ack(int fd, const struct coredump_req *req, char buffer[PAGE_SIZE]; } large_ack = {}; + if (len > sizeof(large_ack)) + return false; + + large_ack.ack = *ack; + ret = send(fd, &large_ack, len, MSG_NOSIGNAL); + if (ret != len) { + fprintf(stderr, "%s: short send %zd: %m\n", __func__, ret); + return false; + } + + fprintf(stderr, "Sent %zu bytes of coredump ack: size %u, mask 0x%llx, types 0x%llx\n", + len, ack->size, (unsigned long long)ack->mask, + (unsigned long long)ack->memory_types); + return true; +} + +bool send_coredump_ack_types(int fd, const struct coredump_req *req, + __u64 mask, __u64 memory_types, size_t size_ack) +{ + struct coredump_ack ack = { + .mask = mask, + .memory_types = memory_types, + }; + if (!size_ack) size_ack = sizeof(struct coredump_ack) < req->size_ack ? sizeof(struct coredump_ack) : req->size_ack; - large_ack.ack.mask = mask; - large_ack.ack.size = size_ack; - ret = send(fd, &large_ack, size_ack, MSG_NOSIGNAL); - if (ret != size_ack) - return false; + ack.size = size_ack; + return send_coredump_ack_bytes(fd, &ack, size_ack); +} - fprintf(stderr, "Sent coredump ack with size %zu and mask 0x%llx\n", - size_ack, (unsigned long long)mask); - return true; +bool send_coredump_ack(int fd, const struct coredump_req *req, + __u64 mask, size_t size_ack) +{ + return send_coredump_ack_types(fd, req, mask, 0, size_ack); } -bool check_coredump_req(const struct coredump_req *req, size_t min_size, - __u64 required_mask) +/* Every option the kernel is expected to advertise in coredump_req->mask. */ +#define TEST_REQ_MASK_ALL \ + (COREDUMP_KERNEL | COREDUMP_USERSPACE | \ + COREDUMP_REJECT | COREDUMP_WAIT | \ + COREDUMP_RECORDS | COREDUMP_SPARSE | COREDUMP_MEMORY_TYPES) + +bool check_coredump_req(const struct coredump_req *req) { - if (req->size < min_size) + if (req->size != COREDUMP_REQ_SIZE_VER1) { + fprintf(stderr, "%s: size %u, expected %d\n", + __func__, req->size, COREDUMP_REQ_SIZE_VER1); + return false; + } + if (req->size_ack != COREDUMP_ACK_SIZE_VER1) { + fprintf(stderr, "%s: size_ack %u, expected %d\n", + __func__, req->size_ack, COREDUMP_ACK_SIZE_VER1); return false; - if ((req->mask & required_mask) != required_mask) + } + if (req->mask != TEST_REQ_MASK_ALL) { + fprintf(stderr, "%s: mask 0x%llx, expected 0x%llx\n", + __func__, (unsigned long long)req->mask, + (unsigned long long)TEST_REQ_MASK_ALL); return false; - if (req->mask & ~required_mask) + } + if (req->memory_types_mask != TEST_MEMORY_ALL) { + fprintf(stderr, "%s: memory_types_mask 0x%llx, expected 0x%llx\n", + __func__, (unsigned long long)req->memory_types_mask, + (unsigned long long)TEST_MEMORY_ALL); return false; + } return true; } @@ -381,3 +1778,304 @@ void process_coredump_worker(int fd_coredump, int fd_peer_pidfd, int fd_core_fil close(fd_coredump); _exit(exit_code); } + +/* + * TIF_NOTIFY_SIGNAL coredump helpers. + * + * __dump_emit() takes anything short of a full write as the end of the + * dump, so every emit that blocks on a full transport can lose the rest + * of it. The NT_FILE note is the one emit that is certain to block, + * because it is the only one larger than the transport, so these helpers + * build a note large enough for that and then raise TIF_NOTIFY_SIGNAL on + * the dumping task while that write is in flight. + */ + +static int io_uring_setup_raw(unsigned int entries, struct io_uring_params *p) +{ + return syscall(__NR_io_uring_setup, entries, p); +} + +/* io_uring reads poll32_events back through swahw32() on big endian. */ +static __u32 notify_poll_mask(__u32 events) +{ +#if defined(__BYTE_ORDER) && __BYTE_ORDER == __BIG_ENDIAN + return __swahw32(events); +#else + return events; +#endif +} + +bool coredump_io_uring_available(void) +{ + struct io_uring_params p = {}; + int fd; + + fd = io_uring_setup_raw(1, &p); + if (fd < 0) + return false; + close(fd); + return true; +} + +/* + * Arm a poll on @fd. io_uring leaves ctx->notify_method at TWA_SIGNAL + * unless the ring asks for SQPOLL or COOP_TASKRUN, so the completion runs + * set_notify_signal() against the task that submitted it. That is us, and + * we are about to become the coredumping task. + */ +static int arm_poll_notify(int trigger_fd) +{ + struct io_uring_params p = {}; + unsigned int *sq_tail, *sq_array; + struct io_uring_sqe *sqes; + size_t sqring_sz; + void *sq; + int ring; + + ring = io_uring_setup_raw(8, &p); + if (ring < 0) + return -1; + + sqring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + sq = mmap(NULL, sqring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, ring, IORING_OFF_SQ_RING); + if (sq == MAP_FAILED) + return -1; + + sqes = mmap(NULL, p.sq_entries * sizeof(*sqes), PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, ring, IORING_OFF_SQES); + if (sqes == MAP_FAILED) + return -1; + + sq_tail = (unsigned int *)((char *)sq + p.sq_off.tail); + sq_array = (unsigned int *)((char *)sq + p.sq_off.array); + + memset(&sqes[0], 0, sizeof(sqes[0])); + sqes[0].opcode = IORING_OP_POLL_ADD; + sqes[0].fd = trigger_fd; + sqes[0].poll32_events = notify_poll_mask(POLLIN); + + sq_array[0] = 0; + __atomic_store_n(sq_tail, 1, __ATOMIC_RELEASE); + + if (syscall(__NR_io_uring_enter, ring, 1, 0, 0, NULL, 0) < 0) + return -1; + + /* Deliberately leaked, we are about to crash. */ + return 0; +} + +/* + * Adjacent mappings with identical flags and contiguous file offsets are + * merged into one VMA, which would collapse NT_FILE back to nothing, so + * alternate the protection to keep every mapping an entry of its own. + * Not PROT_EXEC, /tmp is often mounted noexec. Nothing is ever written + * through these so they get no anon_vma and stay out of the dump itself. + */ +static int make_file_mappings(void) +{ + long pgsz = sysconf(_SC_PAGESIZE); + int fd, i; + + fd = open(NOTIFY_SIGNAL_MAPFILE, + O_RDWR | O_CREAT | O_TRUNC | O_CLOEXEC, 0600); + if (fd < 0) + return 0; + if (ftruncate(fd, (off_t)NOTIFY_SIGNAL_MAP_COUNT * pgsz)) { + close(fd); + return 0; + } + + for (i = 0; i < NOTIFY_SIGNAL_MAP_COUNT; i++) { + int prot = (i & 1) ? PROT_READ : (PROT_READ | PROT_WRITE); + + if (mmap(NULL, pgsz, prot, MAP_PRIVATE, fd, + (off_t)i * pgsz) == MAP_FAILED) + break; + } + close(fd); + return i; +} + +void crashing_child_notify_signal(void) +{ + long pgsz = sysconf(_SC_PAGESIZE); + unsigned char *p; + int trigger_fd; + unsigned long off; + + /* Open the read side first so the reader's open() cannot block. */ + trigger_fd = open(NOTIFY_SIGNAL_TRIGGER, O_RDONLY | O_NONBLOCK | O_CLOEXEC); + + /* Exit rather than crash: a dump without the poll armed proves nothing. */ + if (trigger_fd < 0) + _exit(EXIT_FAILURE); + + if (make_file_mappings() < NOTIFY_SIGNAL_MAP_COUNT) + _exit(EXIT_FAILURE); + + p = mmap(NULL, NOTIFY_SIGNAL_ANON_BYTES, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (p == MAP_FAILED) + _exit(EXIT_FAILURE); + for (off = 0; off < NOTIFY_SIGNAL_ANON_BYTES; off += pgsz) + p[off] = 1; + + if (arm_poll_notify(trigger_fd)) + _exit(EXIT_FAILURE); + + /* crash on purpose */ + *(volatile int *)NULL = 0; + _exit(EXIT_FAILURE); +} + +static int pull_trigger(void) +{ + int fd; + + fd = open(NOTIFY_SIGNAL_TRIGGER, O_WRONLY | O_NONBLOCK | O_CLOEXEC); + if (fd < 0) { + fprintf(stderr, "%s: open failed: %m\n", __func__); + return -1; + } + if (write(fd, "x", 1) != 1) { + fprintf(stderr, "%s: write failed: %m\n", __func__); + close(fd); + return -1; + } + close(fd); + return 0; +} + +/* + * phdr[0] is the PT_NOTE entry: elf_core_dump() emits it right after the + * ELF header. Its p_offset is where the notes begin. + */ +static long long note_offset(const unsigned char *hdr) +{ + ElfW(Phdr) ph; + ElfW(Ehdr) eh; + + memcpy(&eh, hdr, sizeof(eh)); + if (memcmp(eh.e_ident, ELFMAG, SELFMAG) || eh.e_type != ET_CORE) + return -1; + memcpy(&ph, hdr + sizeof(eh), sizeof(ph)); + if (ph.p_type != PT_NOTE) + return -1; + return (long long)ph.p_offset; +} + +/* + * Drain a coredump off @fd, counting what arrives. Once the notes have + * started the kernel is inside the one big note write, so poke the fifo + * the crashing task is polling and stop reading, which keeps the + * transport full and the write blocked with a partial count when the + * wakeup lands. Then carry on to end of file. + * + * What arrives is written to @fd_out when that is not negative. + * Returns the number of bytes received, or -1. Failing to trip the fifo + * is an error too: a dump that was never interrupted proves nothing. + */ +ssize_t recv_coredump_notify_signal(int fd, int fd_out, bool arm) +{ + unsigned char hdr[sizeof(ElfW(Ehdr)) + sizeof(ElfW(Phdr))]; + static char buf[64 << 10]; + long pgsz = sysconf(_SC_PAGESIZE); + long long note_off = 0; + size_t hdrlen = 0; + ssize_t total = 0; + bool armed = false; + + for (;;) { + ssize_t n = read(fd, buf, sizeof(buf)); + + if (n < 0) { + if (errno == EINTR) + continue; + return -1; + } + if (n == 0) + break; + if (fd_out >= 0 && write(fd_out, buf, n) != n) + return -1; + + if (hdrlen < sizeof(hdr)) { + size_t want = sizeof(hdr) - hdrlen; + + if (want > (size_t)n) + want = (size_t)n; + memcpy(hdr + hdrlen, buf, want); + hdrlen += want; + if (hdrlen == sizeof(hdr)) + note_off = note_offset(hdr); + } + + total += n; + + if (arm && !armed && note_off > 0 && + total > note_off + (long long)pgsz) { + if (pull_trigger()) + return -1; + armed = true; + usleep(NOTIFY_SIGNAL_STALL_US); + continue; + } + } + + if (arm && !armed) + return -1; + + return total; +} + +/* + * How large the dump was meant to be. The ELF header and the program + * headers are the first thing emitted, so even a truncated dump says how + * far it should have run: the end is max(p_offset + p_filesz). + * + * That end is exact even when the last segment ends in a hole: + * coredump_write() flushes the pending cprm->to_skip with a final one + * byte emit and __dump_skip() writes zeroes for transports that cannot + * seek, so a whole dump carries every byte the headers promise. + */ +long long coredump_expected_size(const char *path) +{ + ElfW(Phdr) *phdr = NULL; + long long expected = 0; + unsigned int nphdr, i; + ElfW(Ehdr) eh; + int fd; + + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return -1; + if (read(fd, &eh, sizeof(eh)) != sizeof(eh)) + goto err; + if (memcmp(eh.e_ident, ELFMAG, SELFMAG) || eh.e_type != ET_CORE) + goto err; + if (!eh.e_phnum || eh.e_phentsize != sizeof(*phdr)) + goto err; + + nphdr = eh.e_phnum; + phdr = calloc(nphdr, sizeof(*phdr)); + if (!phdr) + goto err; + if (pread(fd, phdr, (size_t)nphdr * sizeof(*phdr), (off_t)eh.e_phoff) != + (ssize_t)((size_t)nphdr * sizeof(*phdr))) + goto err; + + for (i = 0; i < nphdr; i++) { + long long end = (long long)phdr[i].p_offset + + (long long)phdr[i].p_filesz; + if (end > expected) + expected = end; + } + + free(phdr); + close(fd); + return expected; +err: + free(phdr); + close(fd); + return -1; +} diff --git a/tools/testing/selftests/coredump/coredump_test_helpers.h b/tools/testing/selftests/coredump/coredump_test_helpers.h new file mode 100644 index 00000000000000..e7f0998347cade --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_test_helpers.h @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __COREDUMP_TEST_HELPERS_H +#define __COREDUMP_TEST_HELPERS_H + +#include +#include +#include +#include + +#include "../pidfd/pidfd.h" + +#ifndef PAGE_SIZE +#define PAGE_SIZE 4096 +#endif + +#define NUM_THREAD_SPAWN 128 + +/* Size of the mostly unpopulated mapping the sparse coredump test maps. */ +#define SPARSE_MAPPING_SIZE (256 * 1024 * 1024) + +/* A task mapping at least this much is worth a record stream. */ +#define SPARSE_STREAM_THRESHOLD (SPARSE_MAPPING_SIZE / 2) + +/* Size of the shared anonymous mapping the memory types tests map. */ +#define MEMORY_MAPPING_SIZE (4 * 1024 * 1024) + +/* Leave the coredump_filter the crashing child inherited alone. */ +#define FILTER_TASK_INHERIT ((__u64)-1) + +/* Every memory type the kernel is expected to advertise. */ +#define TEST_MEMORY_ALL \ + (COREDUMP_MEMORY_ANON_PRIVATE | COREDUMP_MEMORY_ANON_SHARED | \ + COREDUMP_MEMORY_FILE_PRIVATE | COREDUMP_MEMORY_FILE_SHARED | \ + COREDUMP_MEMORY_ELF_HEADERS | \ + COREDUMP_MEMORY_HUGETLB_PRIVATE | COREDUMP_MEMORY_HUGETLB_SHARED | \ + COREDUMP_MEMORY_DAX_PRIVATE | COREDUMP_MEMORY_DAX_SHARED) + +/* Shared helper function declarations */ +void *do_nothing(void *arg); +void crashing_child(void); +void crashing_child_thread(void); +void crashing_child_sparse(size_t size); +void crashing_child_memory(__u64 task_filter, int fd_addr); +bool find_coredump_segment(int fd, __u64 vaddr, ElfW(Phdr) *segment); +bool sum_coredump_segments(int fd, __u64 *data, __u64 *notes); +bool check_coredump_extent(int fd); +bool peer_coredump_filter(int fd_peer_pidfd, __u64 *memory_types); +ssize_t peer_read_mem(int fd_peer_pidfd, __u64 addr, void *buf, size_t len); +ssize_t recv_coredump_records(int fd_coredump, int fd_core_file, + off_t *coredump_size, bool *truncated, + int fd_peer_pidfd); +ssize_t recv_coredump_compact(int fd_coredump, int fd_object, int fd_reference, + off_t *coredump_size); +ssize_t recv_coredump_bytes(int fd_coredump, int fd_core_file); +ssize_t peer_vm_size(int fd_peer_pidfd); +bool is_elf_core(int fd); +int check_compact_coredump(int fd_object, int fd_reference); +int create_detached_tmpfs(void); +int create_and_listen_unix_socket(const char *path); +bool set_core_pattern(const char *pattern); +int get_peer_pidfd(int fd); +int get_peer_pidfd_thread(int fd); +bool get_pidfd_info(int fd_peer_pidfd, struct pidfd_info *info); + +/* Protocol helper function declarations */ +ssize_t recv_marker(int fd); +bool read_marker(int fd, enum coredump_mark mark); +bool read_hangup(int fd); +bool read_coredump_req(int fd, struct coredump_req *req); +bool read_coredump_req_sized(int fd, struct coredump_req *req, size_t user_size); +bool send_coredump_ack(int fd, const struct coredump_req *req, + __u64 mask, size_t size_ack); +bool send_coredump_ack_types(int fd, const struct coredump_req *req, + __u64 mask, __u64 memory_types, size_t size_ack); +bool send_coredump_ack_bytes(int fd, const struct coredump_ack *ack, size_t len); +bool check_coredump_req(const struct coredump_req *req); +int open_coredump_tmpfile(int fd_tmpfs_detached); +void process_coredump_worker(int fd_coredump, int fd_peer_pidfd, int fd_core_file); + +#endif /* __COREDUMP_TEST_HELPERS_H */ diff --git a/tools/testing/selftests/coredump/coredump_worker_test.c b/tools/testing/selftests/coredump/coredump_worker_test.c new file mode 100644 index 00000000000000..9a4270b65a6e6c --- /dev/null +++ b/tools/testing/selftests/coredump/coredump_worker_test.c @@ -0,0 +1,447 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * A user worker as the coredumping thread. + * + * io-wq workers and SQPOLL threads are threads of the process that never + * return to userspace. They block every signal but SIGKILL and SIGSTOP, + * but a tracer can replace that mask with PTRACE_SETSIGMASK and inject a + * coredump signal. get_signal() then runs vfs_coredump() in the worker. + * The worker's own exit bookkeeping runs only after the dump, so a + * zapped sibling that waits for it in its exit path deadlocks with the + * dumper and the whole thread group is stuck in D state. + * + * Inject SIGSEGV into a chosen thread and require that the thread group + * is gone in bounded time, either because the dump completed or because + * SIGKILL still works. A failure leaves the stuck process behind. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "coredump_test.h" + +#ifndef PTRACE_SETSIGMASK +#define PTRACE_SETSIGMASK 0x420b +#endif + +/* The dump of the tiny child takes well under a second. */ +#define EXIT_TIMEOUT_MS 5000 + +FIXTURE_SETUP(coredump) +{ + FILE *file; + int ret; + + self->pid_coredump_server = -ESRCH; + self->fd_tmpfs_detached = -1; + file = fopen("/proc/sys/kernel/core_pattern", "r"); + ASSERT_NE(NULL, file); + + ret = fread(self->original_core_pattern, 1, sizeof(self->original_core_pattern), file); + ASSERT_TRUE(ret || feof(file)); + ASSERT_LT(ret, sizeof(self->original_core_pattern)); + + self->original_core_pattern[ret] = '\0'; + + ret = fclose(file); + ASSERT_EQ(0, ret); +} + +FIXTURE_TEARDOWN(coredump) +{ + const char *reason; + FILE *file; + int ret; + + file = fopen("/proc/sys/kernel/core_pattern", "w"); + if (!file) { + reason = "Unable to open core_pattern"; + goto fail; + } + + ret = fprintf(file, "%s", self->original_core_pattern); + if (ret < 0) { + reason = "Unable to write to core_pattern"; + goto fail; + } + + ret = fclose(file); + if (ret) { + reason = "Unable to close core_pattern"; + goto fail; + } + + return; +fail: + /* This should never happen */ + fprintf(stderr, "Failed to cleanup coredump test: %s\n", reason); +} + +/* A raw ring, no liburing. */ +struct uring { + int fd; + struct io_uring_params params; + void *sq; + size_t sq_len; + struct io_uring_sqe *sqes; + size_t sqes_len; + unsigned int *sq_tail, *sq_mask, *sq_array; + unsigned int *cq_head, *cq_tail, *cq_mask; + struct io_uring_cqe *cqes; +}; + +static int uring_setup(struct uring *r, unsigned int flags) +{ + size_t cq_len; + + memset(r, 0, sizeof(*r)); + r->params.flags = flags; + if (flags & IORING_SETUP_SQPOLL) + r->params.sq_thread_idle = 2000; + r->fd = syscall(__NR_io_uring_setup, 8, &r->params); + if (r->fd < 0) + return -1; + if (!(r->params.features & IORING_FEAT_SINGLE_MMAP)) + return -1; + + r->sq_len = r->params.sq_off.array + r->params.sq_entries * sizeof(unsigned int); + cq_len = r->params.cq_off.cqes + r->params.cq_entries * sizeof(struct io_uring_cqe); + if (cq_len > r->sq_len) + r->sq_len = cq_len; + r->sq = mmap(NULL, r->sq_len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, r->fd, IORING_OFF_SQ_RING); + if (r->sq == MAP_FAILED) + return -1; + r->sqes_len = r->params.sq_entries * sizeof(struct io_uring_sqe); + r->sqes = mmap(NULL, r->sqes_len, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, r->fd, IORING_OFF_SQES); + if (r->sqes == MAP_FAILED) + return -1; + + r->sq_tail = r->sq + r->params.sq_off.tail; + r->sq_mask = r->sq + r->params.sq_off.ring_mask; + r->sq_array = r->sq + r->params.sq_off.array; + r->cq_head = r->sq + r->params.cq_off.head; + r->cq_tail = r->sq + r->params.cq_off.tail; + r->cq_mask = r->sq + r->params.cq_off.ring_mask; + r->cqes = r->sq + r->params.cq_off.cqes; + return 0; +} + +/* Submit one sqe, wait for its completion and return the result. */ +static int uring_submit_wait(struct uring *r, const struct io_uring_sqe *sqe) +{ + unsigned int tail = *r->sq_tail, idx = tail & *r->sq_mask; + unsigned int flags = IORING_ENTER_GETEVENTS; + int i; + + r->sqes[idx] = *sqe; + r->sq_array[idx] = idx; + __atomic_store_n(r->sq_tail, tail + 1, __ATOMIC_RELEASE); + + if (r->params.flags & IORING_SETUP_SQPOLL) + flags |= IORING_ENTER_SQ_WAKEUP; + + for (i = 0; i < 100; i++) { + if (syscall(__NR_io_uring_enter, r->fd, 1, 1, flags, NULL, 0) < 0 && + errno != EINTR) + return -1; + if (__atomic_load_n(r->cq_tail, __ATOMIC_ACQUIRE) != *r->cq_head) { + unsigned int head = *r->cq_head; + int res = r->cqes[head & *r->cq_mask].res; + + __atomic_store_n(r->cq_head, head + 1, __ATOMIC_RELEASE); + return res; + } + flags &= ~IORING_ENTER_SQ_WAKEUP; + usleep(10 * 1000); + } + return -1; +} + +static bool uring_available(unsigned int flags) +{ + struct io_uring_params params = { .flags = flags }; + int fd; + + fd = syscall(__NR_io_uring_setup, 2, ¶ms); + if (fd < 0) + return false; + close(fd); + return true; +} + +/* + * Keep a ring, an idle io-wq worker and with SQPOLL an SQPOLL thread + * alive. The last worker of a ring never exits on its idle timeout. + */ +static void worker_child(bool sqpoll, int fd_ipc) +{ + struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; + struct io_uring_sqe sqe = {}; + static char buf[64]; + struct uring ring; + int memfd; + + if (setrlimit(RLIMIT_CORE, &rl)) + _exit(EXIT_FAILURE); + + memfd = memfd_create("coredump_worker", 0); + if (memfd < 0 || write(memfd, "hello", 5) != 5) + _exit(EXIT_FAILURE); + + if (uring_setup(&ring, sqpoll ? IORING_SETUP_SQPOLL : 0)) + _exit(EXIT_FAILURE); + + /* IOSQE_ASYNC forces the read through io-wq so a worker appears. */ + sqe.opcode = IORING_OP_READ; + sqe.fd = memfd; + sqe.addr = (__u64)(uintptr_t)buf; + sqe.len = sizeof(buf); + sqe.flags = IOSQE_ASYNC; + if (uring_submit_wait(&ring, &sqe) != 5) + _exit(EXIT_FAILURE); + + if (write_nointr(fd_ipc, "1", 1) != 1) + _exit(EXIT_FAILURE); + close(fd_ipc); + + for (;;) + pause(); +} + +/* Find the thread of @pid whose comm starts with @prefix. */ +static pid_t find_thread(pid_t pid, const char *prefix) +{ + char path[64], comm[64]; + pid_t tid = -1; + struct dirent *de; + ssize_t bytes; + DIR *dir; + int fd; + + snprintf(path, sizeof(path), "/proc/%d/task", pid); + dir = opendir(path); + if (!dir) + return -1; + + while (tid < 0 && (de = readdir(dir))) { + if (!isdigit(de->d_name[0])) + continue; + snprintf(path, sizeof(path), "/proc/%d/task/%s/comm", pid, de->d_name); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + continue; + bytes = read(fd, comm, sizeof(comm) - 1); + close(fd); + if (bytes <= 0) + continue; + comm[bytes] = '\0'; + if (!strncmp(comm, prefix, strlen(prefix))) + tid = atoi(de->d_name); + } + closedir(dir); + return tid; +} + +/* + * Attach, stop the thread with SIGSTOP, drop the signal mask that + * copy_process() gave it and resume it with SIGSEGV. Returns 1 when the + * mask was changed, 0 when PTRACE_SETSIGMASK was refused (a user worker + * keeps its mask and the SIGSEGV stays pending), -1 on any other failure. + */ +static int inject_coredump_signal(pid_t pid, pid_t tid) +{ + __u64 mask = 0; + int status, ret = 1; + + if (ptrace(PTRACE_SEIZE, tid, NULL, NULL)) + return -1; + if (syscall(SYS_tgkill, pid, tid, SIGSTOP)) + return -1; + if (waitpid(tid, &status, __WALL) != tid) + return -1; + if (!WIFSTOPPED(status) || WSTOPSIG(status) != SIGSTOP) + return -1; + if (ptrace(PTRACE_SETSIGMASK, tid, sizeof(mask), &mask)) { + if (errno != EPERM) + return -1; + ret = 0; + } + if (ptrace(PTRACE_DETACH, tid, NULL, (void *)(long)SIGSEGV)) + return -1; + return ret; +} + +/* Reap @pid within @timeout_ms, -1 when it is still there. */ +static int wait_exit(pid_t pid, int *status, int timeout_ms) +{ + int i; + + for (i = 0; i < timeout_ms / 10; i++) { + pid_t ret = waitpid(pid, status, WNOHANG); + + if (ret == pid) + return 0; + if (ret < 0) + return -1; + usleep(10 * 1000); + } + return -1; +} + +static void log_threads(struct __test_metadata *const _metadata, pid_t pid) +{ + char path[64], line[256], comm[64] = {}; + struct dirent *de; + DIR *dir; + FILE *f; + + snprintf(path, sizeof(path), "/proc/%d/task", pid); + dir = opendir(path); + if (!dir) + return; + while ((de = readdir(dir))) { + if (!isdigit(de->d_name[0])) + continue; + snprintf(path, sizeof(path), "/proc/%d/task/%s/status", pid, de->d_name); + f = fopen(path, "r"); + if (!f) + continue; + while (fgets(line, sizeof(line), f)) { + line[strcspn(line, "\n")] = '\0'; + if (!strncmp(line, "Name:", 5)) + snprintf(comm, sizeof(comm), "%s", line + 6); + else if (!strncmp(line, "State:", 6)) + TH_LOG("tid %s (%s) %s", de->d_name, comm, line + 7); + } + fclose(f); + } + closedir(dir); +} + +enum dumper { + DUMPER_MAIN, + DUMPER_WORKER, + DUMPER_SQPOLL, +}; + +static void run_dumper(struct __test_metadata *const _metadata, bool sqpoll, + enum dumper dumper) +{ + bool killed = false; + char path[64], c; + int ipc[2], status, fd, ret; + pid_t pid, tid; + + ASSERT_TRUE(set_core_pattern("/tmp/coredump.file.%p")); + ASSERT_EQ(pipe2(ipc, O_CLOEXEC), 0); + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) { + close(ipc[0]); + worker_child(sqpoll, ipc[1]); + } + close(ipc[1]); + ASSERT_EQ(read_nointr(ipc[0], &c, 1), 1); + close(ipc[0]); + + switch (dumper) { + case DUMPER_MAIN: + tid = pid; + break; + case DUMPER_WORKER: + tid = find_thread(pid, "iou-wrk-"); + break; + case DUMPER_SQPOLL: + tid = find_thread(pid, "iou-sqp-"); + break; + } + ASSERT_GT(tid, 0); + ret = inject_coredump_signal(pid, tid); + ASSERT_GE(ret, 0); + if (!ret) { + /* The signal sits on the worker, the group must be untouched. */ + ASSERT_NE(dumper, DUMPER_MAIN); + TH_LOG("PTRACE_SETSIGMASK refused for tid %d, the SIGSEGV stays pending", tid); + ASSERT_EQ(wait_exit(pid, &status, 1000), -1); + kill(pid, SIGKILL); + ASSERT_EQ(wait_exit(pid, &status, EXIT_TIMEOUT_MS), 0); + ASSERT_TRUE(WIFSIGNALED(status)); + ASSERT_EQ(WTERMSIG(status), SIGKILL); + return; + } + + if (wait_exit(pid, &status, EXIT_TIMEOUT_MS)) { + /* No dump. Whatever happened, SIGKILL must still work. */ + log_threads(_metadata, pid); + kill(pid, SIGKILL); + killed = true; + ASSERT_EQ(wait_exit(pid, &status, EXIT_TIMEOUT_MS), 0) { + TH_LOG("thread group %d is stuck after SIGSEGV to tid %d", + pid, tid); + } + } + + ASSERT_TRUE(WIFSIGNALED(status)); + if (killed) { + TH_LOG("tid %d did not dump, the group was killed instead", tid); + ASSERT_EQ(WTERMSIG(status), SIGKILL); + return; + } + ASSERT_EQ(WTERMSIG(status), SIGSEGV); + ASSERT_TRUE(WCOREDUMP(status)); + + snprintf(path, sizeof(path), "/tmp/coredump.file.%d", pid); + fd = open(path, O_RDONLY | O_CLOEXEC); + unlink(path); + ASSERT_GE(fd, 0); + ASSERT_TRUE(check_coredump_extent(fd)); + close(fd); +} + +/* The mechanics: an injected SIGSEGV into a normal thread dumps core. */ +TEST_F(coredump, main_thread_dumper) +{ + if (!uring_available(0)) + SKIP(return, "io_uring is not available"); + run_dumper(_metadata, false, DUMPER_MAIN); +} + +TEST_F(coredump, plain_worker_dumper) +{ + if (!uring_available(0)) + SKIP(return, "io_uring is not available"); + run_dumper(_metadata, false, DUMPER_WORKER); +} + +TEST_F(coredump, sqpoll_thread_dumper) +{ + if (!uring_available(IORING_SETUP_SQPOLL)) + SKIP(return, "io_uring SQPOLL is not available"); + run_dumper(_metadata, true, DUMPER_SQPOLL); +} + +/* + * The SQPOLL thread leaves its loop on the zap and waits for its io-wq + * workers to exit before it parks. The dumping worker never does. + */ +TEST_F(coredump, sqpoll_worker_dumper) +{ + if (!uring_available(IORING_SETUP_SQPOLL)) + SKIP(return, "io_uring SQPOLL is not available"); + run_dumper(_metadata, true, DUMPER_WORKER); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/exec/Makefile b/tools/testing/selftests/exec/Makefile index b640af8f02b511..2220ed345e920b 100644 --- a/tools/testing/selftests/exec/Makefile +++ b/tools/testing/selftests/exec/Makefile @@ -45,6 +45,10 @@ TEST_GEN_FILES += binfmt_transparent_interp TEST_GEN_PROGS += binfmt_misc_loader TEST_GEN_FILES += binfmt_loader_payload binfmt_loader_payload_static +# Only ASCII punctuation delimits the fields of a register string, so a new +# flag character cannot change which strings register. No bpf toolchain. +TEST_GEN_PROGS += binfmt_misc_delim + # binfmt_misc bpf-backed ('B') handler test: a libbpf harness plus its # struct_ops objects and the test interpreter/app it routes between. Only # built when clang, bpftool, the vmlinux BTF and libbpf are all present diff --git a/tools/testing/selftests/exec/binfmt_misc_delim.c b/tools/testing/selftests/exec/binfmt_misc_delim.c new file mode 100644 index 00000000000000..ffc17cb7854502 --- /dev/null +++ b/tools/testing/selftests/exec/binfmt_misc_delim.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test which characters may delimit the fields of a register string. + */ +#define _GNU_SOURCE +#include +#include + +#include "binfmt_misc_common.h" +#include "kselftest_harness.h" + +#define ENTRY "bmdelim" +/* Shares no character with the sets below, or a refusal proves nothing. */ +#define MAGIC "bmmagic" +#define INTERP "/bin/true" + +/* + * ASCII punctuation without '\' and '/'. The backslash is refused because + * it would cut a magic that uses \x to escape short. '/' is accepted + * but cannot delimit a rule that names an absolute interpreter. + */ +#define PUNCTUATION "!\"#$%&'()*+,-.:;<=>?@[]^_`{|}~" + +/* 'M', 'E' and 'B' name types, 'P' through 'D' are the flags. */ +#define LETTERS "MEBPOCFTLDqz" +#define DIGITS "0157" +#define WHITESPACE " \t\n" +#define CONTROL "\001\033\177" +#define NON_ASCII "\200\244\377" + +/* ':bmdelim:E::bmmagic::/bin/true:' with @del in place of every ':'. */ +static int register_with(char del) +{ + char rule[128]; + + snprintf(rule, sizeof(rule), "%c%s%cE%c%c%s%c%c%s%c", del, ENTRY, del, + del, del, MAGIC, del, del, INTERP, del); + return write_reg(rule); +} + +/* No character of @set may delimit a register string. */ +static void expect_refused(struct __test_metadata *_metadata, const char *set) +{ + const char *d; + + for (d = set; *d; d++) { + int rc = register_with(*d); + + EXPECT_EQ(rc, -1) + TH_LOG("%#x delimited a register string", + (unsigned char)*d); + if (rc == 0) { + unregister(ENTRY); + continue; + } + EXPECT_EQ(errno, EINVAL); + } +} + +FIXTURE(delim) { +}; + +FIXTURE_SETUP(delim) +{ + if (getuid() != 0) + SKIP(return, "test must be run as root"); + if (!binfmt_misc_available()) + SKIP(return, "no binfmt_misc"); + + /* A kernel without the allow-list takes any character but a flag. */ + if (register_with('q') == 0) { + unregister(ENTRY); + SKIP(return, "kernel without the delimiter allow-list"); + } +} + +FIXTURE_TEARDOWN(delim) +{ + unregister(ENTRY); +} + +/* Punctuation delimits, which is all anything deployed ever uses. */ +TEST_F(delim, punctuation_accepted) +{ + const char *d; + + for (d = PUNCTUATION; *d; d++) { + EXPECT_EQ(register_with(*d), 0) + TH_LOG("'%c' refused with errno %d", *d, errno); + unregister(ENTRY); + } +} + +/* Letters name the types and the flags, so none of them can delimit. */ +TEST_F(delim, letters_refused) +{ + expect_refused(_metadata, LETTERS); +} + +/* The offset field is written in digits. */ +TEST_F(delim, digits_refused) +{ + expect_refused(_metadata, DIGITS); +} + +TEST_F(delim, whitespace_refused) +{ + expect_refused(_metadata, WHITESPACE); +} + +TEST_F(delim, control_refused) +{ + expect_refused(_metadata, CONTROL); +} + +TEST_F(delim, non_ascii_refused) +{ + expect_refused(_metadata, NON_ASCII); +} + +/* The escape character would cut every magic that uses one short. */ +TEST_F(delim, backslash_refused) +{ + expect_refused(_metadata, "\\"); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/filesystems/.gitignore b/tools/testing/selftests/filesystems/.gitignore index 62f7b1c4664970..cf79000e4092a1 100644 --- a/tools/testing/selftests/filesystems/.gitignore +++ b/tools/testing/selftests/filesystems/.gitignore @@ -2,7 +2,6 @@ dnotify_test devpts_pts fclog -file_stressor anon_inode_test kernfs_test idmapped_tmpfile diff --git a/tools/testing/selftests/filesystems/Makefile b/tools/testing/selftests/filesystems/Makefile index 93e2cc9123b041..fbd5c27505ef58 100644 --- a/tools/testing/selftests/filesystems/Makefile +++ b/tools/testing/selftests/filesystems/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 CFLAGS += $(KHDR_INCLUDES) -TEST_GEN_PROGS := devpts_pts file_stressor anon_inode_test kernfs_test fclog ustat_test rw_hint_test +TEST_GEN_PROGS := devpts_pts anon_inode_test kernfs_test fclog ustat_test rw_hint_test TEST_GEN_PROGS += idmapped_tmpfile TEST_GEN_PROGS_EXTENDED := dnotify_test diff --git a/tools/testing/selftests/filesystems/config b/tools/testing/selftests/filesystems/config new file mode 100644 index 00000000000000..9f45bc493a30a8 --- /dev/null +++ b/tools/testing/selftests/filesystems/config @@ -0,0 +1,8 @@ +CONFIG_CGROUPS=y +CONFIG_CGROUP_PIDS=y +CONFIG_FHANDLE=y +CONFIG_NAMESPACES=y +CONFIG_NET=y +CONFIG_NET_NS=y +CONFIG_INET=y +CONFIG_SYSFS=y diff --git a/tools/testing/selftests/filesystems/configfs/.gitignore b/tools/testing/selftests/filesystems/configfs/.gitignore new file mode 100644 index 00000000000000..accfb6bb48268c --- /dev/null +++ b/tools/testing/selftests/filesystems/configfs/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +configfs_test diff --git a/tools/testing/selftests/filesystems/configfs/Makefile b/tools/testing/selftests/filesystems/configfs/Makefile new file mode 100644 index 00000000000000..40a91ed788ed2a --- /dev/null +++ b/tools/testing/selftests/filesystems/configfs/Makefile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2026 Meta Platforms, Inc. and affiliates +# Copyright (c) 2026 Breno Leitao + +CFLAGS += -Wall -Werror -pthread +TEST_GEN_PROGS := configfs_test + +include ../../lib.mk diff --git a/tools/testing/selftests/filesystems/configfs/config b/tools/testing/selftests/filesystems/configfs/config new file mode 100644 index 00000000000000..5ea17df535b3fd --- /dev/null +++ b/tools/testing/selftests/filesystems/configfs/config @@ -0,0 +1,5 @@ +CONFIG_CONFIGFS_FS=y +CONFIG_MODULES=y +CONFIG_MODULE_UNLOAD=y +CONFIG_SAMPLES=y +CONFIG_SAMPLE_CONFIGFS=m diff --git a/tools/testing/selftests/filesystems/configfs/configfs_test.c b/tools/testing/selftests/filesystems/configfs/configfs_test.c new file mode 100644 index 00000000000000..c6a1049e5852e5 --- /dev/null +++ b/tools/testing/selftests/filesystems/configfs/configfs_test.c @@ -0,0 +1,481 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Exercise the configfs userspace interface through the subsystems + * registered by samples/configfs. + * + * Copyright (c) 2026 Meta Platforms, Inc. and affiliates + * Copyright (c) 2026 Breno Leitao + */ +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kselftest_harness.h" + +/* Private to fs/configfs/mount.c. */ +#define CONFIGFS_MAGIC 0x62656570 + +#define SAMPLE_MODULE "configfs_sample" + +#define CHILDLESS "01-childless" +#define SIMPLE "02-simple-children" +#define GROUPS "03-group-children" +#define SYMLINKS "04-symlink-children" + +#define ITEM_A SIMPLE "/kselftest-a" +#define ITEM_B SIMPLE "/kselftest-b" +#define GROUP GROUPS "/kselftest-group" +#define GROUP_ITEM GROUP "/kselftest-a" +#define LINK_SRC SYMLINKS "/kselftest-src" +#define LINK LINK_SRC "/kselftest-link" + +#define RACE_ITERATIONS 20000 + +static const char * const test_links[] = { + LINK, +}; + +/* Deepest first, so one pass empties the tree. */ +static const char * const test_dirs[] = { + GROUP_ITEM, + GROUP, + LINK_SRC, + ITEM_A, + ITEM_B, +}; + +static void drop_test_dirs(void) +{ + size_t i; + + /* Links first: they hold both their source and their target. */ + for (i = 0; i < ARRAY_SIZE(test_links); i++) + unlink(test_links[i]); + + for (i = 0; i < ARRAY_SIZE(test_dirs); i++) + rmdir(test_dirs[i]); +} + +static ssize_t read_attr(const char *path, char *buf, size_t len) +{ + ssize_t ret; + int fd; + + fd = open(path, O_RDONLY); + if (fd < 0) + return -1; + + ret = read(fd, buf, len - 1); + close(fd); + if (ret < 0) + return -1; + + buf[ret] = '\0'; + return ret; +} + +static ssize_t write_attr(const char *path, const char *val) +{ + ssize_t ret; + int fd, err; + + fd = open(path, O_WRONLY); + if (fd < 0) + return -1; + + ret = write(fd, val, strlen(val)); + err = errno; + close(fd); + errno = err; + + return ret; +} + +FIXTURE(configfs) { + char mnt[sizeof(P_tmpdir "/configfs_XXXXXX")]; + bool mounted; +}; + +FIXTURE_SETUP(configfs) +{ + char tmpl[] = P_tmpdir "/configfs_XXXXXX"; + + if (geteuid()) + SKIP(return, "need root to load modules and mount configfs"); + + ASSERT_EQ(system("modprobe -q " SAMPLE_MODULE), 0) + TH_LOG(SAMPLE_MODULE " missing, is CONFIG_SAMPLE_CONFIGFS=m?"); + + ASSERT_EQ(unshare(CLONE_NEWNS), 0); + ASSERT_EQ(mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL), 0); + + ASSERT_NE(mkdtemp(tmpl), NULL); + strcpy(self->mnt, tmpl); + + ASSERT_EQ(mount("configfs", self->mnt, "configfs", 0, NULL), 0); + ASSERT_EQ(chdir(self->mnt), 0); + self->mounted = true; + + /* configfs items outlive the mount, so a killed run leaves some. */ + drop_test_dirs(); +} + +FIXTURE_TEARDOWN(configfs) +{ + if (self->mounted) { + drop_test_dirs(); + EXPECT_EQ(chdir("/"), 0); + EXPECT_EQ(umount2(self->mnt, MNT_DETACH), 0); + } + + if (self->mnt[0]) + EXPECT_EQ(rmdir(self->mnt), 0); +} + +TEST_F(configfs, mount_and_subsystems) +{ + const char * const subsys[] = { CHILDLESS, SIMPLE, GROUPS, SYMLINKS }; + struct statfs sfs; + struct stat st; + size_t i; + + ASSERT_EQ(statfs(".", &sfs), 0); + EXPECT_EQ(sfs.f_type, CONFIGFS_MAGIC); + + for (i = 0; i < ARRAY_SIZE(subsys); i++) { + ASSERT_EQ(stat(subsys[i], &st), 0) + TH_LOG("%s is missing", subsys[i]); + EXPECT_TRUE(S_ISDIR(st.st_mode)); + } +} + +TEST_F(configfs, mkdir_at_root) +{ + /* The root has no ->mkdir(); only subsystems register there. */ + ASSERT_EQ(mkdir("kselftest-root", 0755), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, rmdir_subsystem) +{ + ASSERT_EQ(rmdir(CHILDLESS), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, mkdir_without_group_ops) +{ + /* 01-childless has attributes but no ->make_item()/->make_group(). */ + ASSERT_EQ(mkdir(CHILDLESS "/kselftest-a", 0755), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, attr_store_and_show) +{ + char buf[64]; + + ASSERT_GT(write_attr(CHILDLESS "/storeme", "42"), 0); + ASSERT_GT(read_attr(CHILDLESS "/storeme", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "42\n"); +} + +TEST_F(configfs, attr_store_rejects_garbage) +{ + ASSERT_EQ(write_attr(CHILDLESS "/storeme", "not-a-number"), -1); + EXPECT_EQ(errno, EINVAL); +} + +TEST_F(configfs, attr_show_runs_on_every_open) +{ + char first[64], second[64]; + + /* 01-childless/showme increments the value it just returned. */ + ASSERT_GT(read_attr(CHILDLESS "/showme", first, sizeof(first)), 0); + ASSERT_GT(read_attr(CHILDLESS "/showme", second, sizeof(second)), 0); + EXPECT_EQ(atoi(second), atoi(first) + 1); +} + +TEST_F(configfs, attr_read_only) +{ + ASSERT_EQ(open(CHILDLESS "/description", O_WRONLY), -1); + EXPECT_EQ(errno, EACCES); +} + +TEST_F(configfs, attr_unlink) +{ + /* ->unlink() only accepts the symlinks configfs itself created. */ + ASSERT_EQ(unlink(CHILDLESS "/storeme"), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, attr_read_length) +{ + char buf[8192]; + struct stat st; + ssize_t n; + int fd; + + fd = open(CHILDLESS "/description", O_RDONLY); + ASSERT_GE(fd, 0); + + /* Attributes report a page, whatever ->show() ends up producing. */ + ASSERT_EQ(fstat(fd, &st), 0); + EXPECT_EQ(st.st_size, sysconf(_SC_PAGESIZE)); + + n = read(fd, buf, sizeof(buf)); + ASSERT_GT(n, 0); + EXPECT_LT(n, st.st_size); + EXPECT_EQ(read(fd, buf, sizeof(buf)), 0); + + EXPECT_EQ(close(fd), 0); +} + +TEST_F(configfs, attr_write_is_not_incremental) +{ + char buf[64]; + int fd; + + /* + * Every write hands the whole buffer to ->store() and the file + * position is ignored, so the second write replaces the first. + */ + fd = open(CHILDLESS "/storeme", O_WRONLY); + ASSERT_GE(fd, 0); + ASSERT_EQ(write(fd, "1", 1), 1); + ASSERT_EQ(write(fd, "2", 1), 1); + EXPECT_EQ(close(fd), 0); + + ASSERT_GT(read_attr(CHILDLESS "/storeme", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "2\n"); +} + +TEST_F(configfs, item_create_and_drop) +{ + struct stat st; + + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + EXPECT_EQ(stat(ITEM_A "/storeme", &st), 0); + + /* The item carries its own attributes, not the subsystem's. */ + ASSERT_EQ(stat(ITEM_A "/description", &st), -1); + EXPECT_EQ(errno, ENOENT); + + ASSERT_EQ(rmdir(ITEM_A), 0); + ASSERT_EQ(stat(ITEM_A, &st), -1); + EXPECT_EQ(errno, ENOENT); +} + +TEST_F(configfs, item_create_twice) +{ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(mkdir(ITEM_A, 0755), -1); + EXPECT_EQ(errno, EEXIST); +} + +TEST_F(configfs, item_has_no_children) +{ + /* ->make_item() produces an item, so it cannot nest. */ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(mkdir(ITEM_A "/kselftest-b", 0755), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, item_attrs_are_private) +{ + char buf[64]; + + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(mkdir(ITEM_B, 0755), 0); + + ASSERT_GT(write_attr(ITEM_A "/storeme", "11"), 0); + ASSERT_GT(write_attr(ITEM_B "/storeme", "22"), 0); + + ASSERT_GT(read_attr(ITEM_A "/storeme", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "11\n"); + ASSERT_GT(read_attr(ITEM_B "/storeme", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "22\n"); +} + +TEST_F(configfs, group_create_and_drop) +{ + struct stat st; + + /* 03-group-children hands out groups that take items of their own. */ + ASSERT_EQ(mkdir(GROUP, 0755), 0); + EXPECT_EQ(stat(GROUP "/description", &st), 0); + + ASSERT_EQ(mkdir(GROUP_ITEM, 0755), 0); + EXPECT_EQ(stat(GROUP_ITEM "/storeme", &st), 0); + + ASSERT_EQ(rmdir(GROUP), -1); + EXPECT_EQ(errno, ENOTEMPTY); + + ASSERT_EQ(rmdir(GROUP_ITEM), 0); + ASSERT_EQ(rmdir(GROUP), 0); +} + +TEST_F(configfs, rename_item) +{ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(rename(ITEM_A, ITEM_B), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, symlink_without_allow_link) +{ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(symlink(ITEM_A, SIMPLE "/kselftest-link"), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, symlink_and_unlink) +{ + char buf[PATH_MAX]; + struct stat st; + ssize_t n; + + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + + ASSERT_EQ(symlink(ITEM_A, LINK), 0); + + /* configfs stores its own body, a path relative to the link. */ + n = readlink(LINK, buf, sizeof(buf) - 1); + ASSERT_GT(n, 0); + buf[n] = '\0'; + EXPECT_STREQ(buf, "../../" ITEM_A); + EXPECT_EQ(stat(LINK "/storeme", &st), 0); + + /* ->allow_link() ran on the source, not on the target. */ + ASSERT_GT(read_attr(LINK_SRC "/nlinks", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "1\n"); + + ASSERT_EQ(unlink(LINK), 0); + ASSERT_GT(read_attr(LINK_SRC "/nlinks", buf, sizeof(buf)), 0); + EXPECT_STREQ(buf, "0\n"); +} + +TEST_F(configfs, symlink_pins_both_ends) +{ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + ASSERT_EQ(symlink(ITEM_A, LINK), 0); + + /* A linked item cannot go away under the link. */ + ASSERT_EQ(rmdir(ITEM_A), -1); + EXPECT_EQ(errno, EBUSY); + + /* The link counts as a child of its source. */ + ASSERT_EQ(rmdir(LINK_SRC), -1); + EXPECT_EQ(errno, ENOTEMPTY); + + ASSERT_EQ(unlink(LINK), 0); + EXPECT_EQ(rmdir(ITEM_A), 0); +} + +TEST_F(configfs, symlink_target_outside_configfs) +{ + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + ASSERT_EQ(symlink("/", LINK), -1); + EXPECT_EQ(errno, EPERM); +} + +TEST_F(configfs, symlink_target_missing) +{ + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + ASSERT_EQ(symlink(SIMPLE "/kselftest-gone", LINK), -1); + EXPECT_EQ(errno, ENOENT); +} + +TEST_F(configfs, symlink_target_is_an_attribute) +{ + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + + /* The target is resolved with LOOKUP_DIRECTORY. */ + ASSERT_EQ(symlink(CHILDLESS "/storeme", LINK), -1); + EXPECT_EQ(errno, ENOTDIR); +} + +static volatile int race_stop; + +static void *rmdir_target(void *arg) +{ + while (!race_stop) { + if (mkdir(ITEM_A, 0755) == 0 || errno == EEXIST) + rmdir(ITEM_A); + } + + return NULL; +} + +/* -1 if the kernel does not export a warning counter. */ +static long warn_count(void) +{ + char buf[32]; + + if (read_attr("/sys/kernel/warn_count", buf, sizeof(buf)) < 0) + return -1; + + return strtol(buf, NULL, 10); +} + +TEST_F(configfs, symlink_races_with_target_rmdir) +{ + pthread_t thread; + long warns; + int i; + + warns = warn_count(); + if (warns < 0) + SKIP(return, "no /sys/kernel/warn_count to watch"); + + ASSERT_EQ(mkdir(LINK_SRC, 0755), 0); + ASSERT_EQ(pthread_create(&thread, NULL, rmdir_target, NULL), 0); + + /* + * configfs_rmdir() drops the last reference to the item while its + * dentry is still hashed, and get_target() takes a hashed dentry as + * proof that the item behind it is alive. The symlink then walks + * ->ci_dentry into a released dirent, which configfs_get() warns + * about. KASAN sees the freed item itself. + */ + for (i = 0; i < RACE_ITERATIONS; i++) { + if (symlink(ITEM_A, LINK) == 0) + unlink(LINK); + + /* Give up on the first splat rather than flood the log. */ + if (!(i % 128) && warn_count() != warns) + break; + } + + race_stop = 1; + ASSERT_EQ(pthread_join(thread, NULL), 0); + + EXPECT_EQ(warn_count(), warns) + TH_LOG("kernel warned after %d iterations", i); +} + +TEST_F(configfs, module_pinned_by_item) +{ + ASSERT_EQ(mkdir(ITEM_A, 0755), 0); + + /* mkdir() pins both the subsystem's module and the new item's. */ + ASSERT_EQ(syscall(__NR_delete_module, SAMPLE_MODULE, O_NONBLOCK), -1); + if (errno == ENOSYS) + SKIP(return, "kernel built without CONFIG_MODULE_UNLOAD"); + EXPECT_EQ(errno, EWOULDBLOCK); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/filesystems/file_stressor/.gitignore b/tools/testing/selftests/filesystems/file_stressor/.gitignore new file mode 100644 index 00000000000000..1eb3f40077d3f1 --- /dev/null +++ b/tools/testing/selftests/filesystems/file_stressor/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +file_stressor diff --git a/tools/testing/selftests/filesystems/file_stressor/Makefile b/tools/testing/selftests/filesystems/file_stressor/Makefile new file mode 100644 index 00000000000000..88c8231ac14435 --- /dev/null +++ b/tools/testing/selftests/filesystems/file_stressor/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 + +CFLAGS += $(KHDR_INCLUDES) +TEST_GEN_PROGS := file_stressor + +include ../../lib.mk diff --git a/tools/testing/selftests/filesystems/file_stressor.c b/tools/testing/selftests/filesystems/file_stressor/file_stressor.c similarity index 100% rename from tools/testing/selftests/filesystems/file_stressor.c rename to tools/testing/selftests/filesystems/file_stressor/file_stressor.c diff --git a/tools/testing/selftests/filesystems/file_stressor/settings b/tools/testing/selftests/filesystems/file_stressor/settings new file mode 100644 index 00000000000000..b675ca93f93691 --- /dev/null +++ b/tools/testing/selftests/filesystems/file_stressor/settings @@ -0,0 +1,3 @@ +# Timeout for file_stressor test +# The test runs for 900 seconds (15 minutes) plus setup/teardown time +timeout=1800 diff --git a/tools/testing/selftests/filesystems/fscontext_ns/.gitignore b/tools/testing/selftests/filesystems/fscontext_ns/.gitignore new file mode 100644 index 00000000000000..a632905257a335 --- /dev/null +++ b/tools/testing/selftests/filesystems/fscontext_ns/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +fscontext_ns_test diff --git a/tools/testing/selftests/filesystems/idmapped_tmpfile.c b/tools/testing/selftests/filesystems/idmapped_tmpfile.c index bc411ab8281eae..77b96d09435785 100644 --- a/tools/testing/selftests/filesystems/idmapped_tmpfile.c +++ b/tools/testing/selftests/filesystems/idmapped_tmpfile.c @@ -91,6 +91,13 @@ FIXTURE_SETUP(idmapped_tmpfile) ASSERT_EQ(mkdir(self->dir, 0777), 0); /* World-writable so an unmapped caller still passes permission(). */ ASSERT_EQ(chmod(self->dir, 0777), 0); + /* + * The directory itself must be inside the range the mount maps. + * Left owned by 0 it has no mapping in the mount, and + * inode_permission() then fails MAY_WRITE via HAS_UNMAPPED_ID() + * before any of the tmpfile logic is reached, regardless of mode. + */ + ASSERT_EQ(chown(self->dir, MAP_HOST, MAP_HOST), 0); } FIXTURE_TEARDOWN(idmapped_tmpfile) diff --git a/tools/testing/selftests/filesystems/kernfs_test.c b/tools/testing/selftests/filesystems/kernfs_test.c index 84c2b910a60db0..f4bcf3caf1d5de 100644 --- a/tools/testing/selftests/filesystems/kernfs_test.c +++ b/tools/testing/selftests/filesystems/kernfs_test.c @@ -2,9 +2,23 @@ #define _GNU_SOURCE #define __SANE_USERSPACE_TYPES__ +#include +#include #include +#include +#include +#include +#include #include +#include +#include +#include +#include +#include +#include +#include #include +#include #include #include "kselftest_harness.h" @@ -12,12 +26,24 @@ TEST(kernfs_listxattr) { + ssize_t len; int fd; - /* Read-only file that can never have any extended attributes set. */ + /* Read-only file that can never have any extended attributes set. + * However, on systems with SELinux enabled, security.selinux xattr + * may be present. Skip the content check if any xattrs are found. + */ fd = open("/sys/kernel/warn_count", O_RDONLY | O_CLOEXEC); ASSERT_GE(fd, 0); - ASSERT_EQ(flistxattr(fd, NULL, 0), 0); + + len = flistxattr(fd, NULL, 0); + ASSERT_GE(len, 0); + + if (len > 0) { + close(fd); + SKIP(return, "xattrs present on /sys/kernel/warn_count, skipping xattr content check"); + } + EXPECT_EQ(close(fd), 0); } @@ -34,5 +60,1176 @@ TEST(kernfs_getxattr) EXPECT_EQ(close(fd), 0); } -TEST_HARNESS_MAIN +/* + * Exercise the kernfs dentry cache: lookup, revalidation of positive and + * negative dentries, readdir and namespace tagging. + * + * These drive kernfs from kernel context rather than VFS create/unlink, + * which is what ->d_revalidate() exists for: writing cgroup.subtree_control + * adds and removes files in every child cgroup with no VFS operation + * touching those names. + */ + +#define CG_SCRATCH "kernfs_selftest" +#define TEST_IFNAME "kfstest0" + +/* + * Controllers that add a file to each child cgroup when enabled. The probe + * file must be owned by the controller: cgroup_base_files[] entries such as + * cpu.stat exist in every cgroup regardless, and every file the cpu + * controller does own is behind a Kconfig symbol, so cpu is not usable here. + */ +static const struct { + const char *name; + const char *probe_file; +} controllers[] = { + { "memory", "memory.current" }, + { "pids", "pids.current" }, +}; + +static int find_cgroup2_root(char *buf, size_t len) +{ + char line[PATH_MAX * 2]; + FILE *f; + int ret = -1; + + f = fopen("/proc/self/mounts", "re"); + if (!f) + return -1; + + while (fgets(line, sizeof(line), f)) { + char mnt[PATH_MAX], type[64]; + + /* Octal escaping can expand a path fourfold; bound both %s. */ + if (sscanf(line, "%*s %4095s %63s", mnt, type) != 2) + continue; + if (strcmp(type, "cgroup2")) + continue; + if (strlen(mnt) >= len) + break; + strcpy(buf, mnt); + ret = 0; + break; + } + + fclose(f); + return ret; +} + +static int write_file(const char *path, const char *val) +{ + ssize_t len = strlen(val); + int fd, ret; + + fd = open(path, O_WRONLY | O_CLOEXEC); + if (fd < 0) + return -1; + ret = write(fd, val, len) == len ? 0 : -1; + close(fd); + return ret; +} + +static bool file_has_word(const char *path, const char *word) +{ + char buf[4096], *tok, *save; + bool found = false; + ssize_t n; + int fd; + + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return false; + n = read(fd, buf, sizeof(buf) - 1); + close(fd); + if (n < 0) + return false; + buf[n] = '\0'; + + for (tok = strtok_r(buf, "\n ", &save); tok; + tok = strtok_r(NULL, "\n ", &save)) { + if (!strcmp(tok, word)) { + found = true; + break; + } + } + return found; +} + +static bool path_is_mounted(const char *path) +{ + char line[PATH_MAX * 2]; + bool found = false; + FILE *f; + + f = fopen("/proc/self/mounts", "re"); + if (!f) + return false; + while (fgets(line, sizeof(line), f)) { + char mnt[PATH_MAX]; + + if (sscanf(line, "%*s %4095s", mnt) != 1) + continue; + if (!strcmp(mnt, path)) { + found = true; + break; + } + } + fclose(f); + return found; +} + +/* Shared by the stress tests below. */ +static bool stress_deadline(const struct timespec *end) +{ + struct timespec now; + + clock_gettime(CLOCK_MONOTONIC, &now); + return now.tv_sec > end->tv_sec || + (now.tv_sec == end->tv_sec && now.tv_nsec >= end->tv_nsec); +} + +FIXTURE(kernfs_cgroup) +{ + char scratch[PATH_MAX]; /* /kernfs_selftest. */ + char child[PATH_MAX]; /* /child */ + char probe[PATH_MAX]; /* child's controller file */ + char scratch_sc[PATH_MAX]; /* scratch's cgroup.subtree_control */ + char root_sc[PATH_MAX]; /* root's cgroup.subtree_control */ + char enable[32]; /* "+" */ + char disable[32]; /* "-" */ + char mnt[PATH_MAX]; /* our own mount, if we made one */ + bool mounted; + bool enabled_at_root; +}; + +/* A cgroup stays busy briefly after its last task exits. */ +static void rmdir_retry(const char *path) +{ + int i; + + for (i = 0; i < 500; i++) { + if (!rmdir(path) || errno != EBUSY) + return; + usleep(10000); + } +} + +/* + * Undo whatever SETUP managed to do. The harness skips TEARDOWN after a + * failed or skipped SETUP, so SETUP must call this before returning early. + */ +static void kernfs_cgroup_undo(FIXTURE_DATA(kernfs_cgroup) *self) +{ + rmdir_retry(self->child); + rmdir(self->scratch); + if (self->enabled_at_root) + write_file(self->root_sc, self->disable); + if (self->mounted) { + umount2(self->mnt, MNT_DETACH); + rmdir(self->mnt); + } + self->enabled_at_root = false; + self->mounted = false; +} + +FIXTURE_SETUP(kernfs_cgroup) +{ + char root[PATH_MAX], ctl[PATH_MAX]; + const char *probe_file = NULL; + size_t i; + + if (geteuid()) + SKIP(return, "test needs to run as root"); + + /* + * A private mount namespace stops our mounts leaking, but does not + * isolate the cgroup hierarchy: cgroup2 has one default hierarchy + * however many times it is mounted. The scratch cgroups live in the + * host's and must be removed, not discarded with the namespace. + */ + if (unshare(CLONE_NEWNS)) + SKIP(return, "unshare(CLONE_NEWNS): %s", strerror(errno)); + if (mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL)) + SKIP(return, "make / private: %s", strerror(errno)); + + /* Use an existing cgroup2 mount if there is one, else make our own. */ + if (find_cgroup2_root(root, sizeof(root))) { + strcpy(self->mnt, "/tmp/kernfs_selftest_cg2.XXXXXX"); + if (!mkdtemp(self->mnt)) + SKIP(return, "mkdtemp: %s", strerror(errno)); + if (mount("none", self->mnt, "cgroup2", 0, NULL)) { + rmdir(self->mnt); + SKIP(return, "mount cgroup2: %s", strerror(errno)); + } + self->mounted = true; + strcpy(root, self->mnt); + } + + snprintf(self->root_sc, sizeof(self->root_sc), + "%s/cgroup.subtree_control", root); + snprintf(ctl, sizeof(ctl), "%s/cgroup.controllers", root); + + /* Named after our pid so we cannot collide with anything else. */ + snprintf(self->scratch, sizeof(self->scratch), "%s/%s.%d", root, + CG_SCRATCH, getpid()); + snprintf(self->child, sizeof(self->child), "%s/child", self->scratch); + snprintf(self->scratch_sc, sizeof(self->scratch_sc), + "%s/cgroup.subtree_control", self->scratch); + + for (i = 0; i < ARRAY_SIZE(controllers); i++) { + if (!file_has_word(ctl, controllers[i].name)) + continue; + snprintf(self->enable, sizeof(self->enable), "+%s", + controllers[i].name); + snprintf(self->disable, sizeof(self->disable), "-%s", + controllers[i].name); + probe_file = controllers[i].probe_file; + + /* + * A controller must be in the root's subtree_control before + * it appears in our scratch cgroup. Note if we enabled it, + * so it can be put back. + */ + self->enabled_at_root = !file_has_word(self->root_sc, + controllers[i].name); + if (self->enabled_at_root && + write_file(self->root_sc, self->enable)) { + self->enabled_at_root = false; + probe_file = NULL; + continue; + } + break; + } + if (!probe_file) { + kernfs_cgroup_undo(self); + SKIP(return, "no usable cgroup2 controller"); + } + + snprintf(self->probe, sizeof(self->probe), "%s/%s", self->child, + probe_file); + + /* + * Only an unusable environment may skip. A scratch cgroup named + * after our own pid should always be creatable, so failing to make + * one is a result -- skipping would let a broken kernel look green. + */ + if (mkdir(self->scratch, 0755)) { + int err = errno; + + kernfs_cgroup_undo(self); + if (err == EROFS || err == EACCES || err == EPERM) + SKIP(return, "mkdir %s: %s", self->scratch, + strerror(err)); + ASSERT_EQ(err, 0) TH_LOG("mkdir %s: %s", self->scratch, + strerror(err)); + } + if (mkdir(self->child, 0755)) { + int err = errno; + + kernfs_cgroup_undo(self); + ASSERT_EQ(err, 0) TH_LOG("mkdir %s: %s", self->child, + strerror(err)); + } + + /* + * The tests below need the probe file to appear and disappear with + * the controller, so it must be absent now, before anything enables + * it in the scratch cgroup. Skip rather than fail: a probe file that + * is already there means the table names one the controller does not + * own, not that the kernel is broken. + */ + if (!access(self->probe, F_OK)) { + kernfs_cgroup_undo(self); + SKIP(return, "%s is not owned by the %s controller", + probe_file, self->enable + 1); + } +} + +FIXTURE_TEARDOWN(kernfs_cgroup) +{ + write_file(self->scratch_sc, self->disable); + kernfs_cgroup_undo(self); +} + +/* + * Walking already-cached dentries must not invalidate them. Spurious + * invalidation is not merely slow: d_invalidate() calls detach_mounts(), so + * an unrelated lookup would silently tear down any mount below. + */ +TEST_F(kernfs_cgroup, path_walk_does_not_invalidate) +{ + char src[] = "/tmp/kernfs_selftest_bind.XXXXXX"; + char sub[PATH_MAX], probe[PATH_MAX]; + int i; + + snprintf(sub, sizeof(sub), "%s/sub", self->child); + ASSERT_EQ(mkdir(sub, 0755), 0); + + if (!mkdtemp(src)) { + rmdir(sub); + SKIP(return, "mkdtemp: %s", strerror(errno)); + } + if (mount(src, sub, NULL, MS_BIND, NULL)) { + int err = errno; + + rmdir(sub); + rmdir(src); + SKIP(return, "bind mount onto a cgroup dir: %s", strerror(err)); + } + ASSERT_TRUE(path_is_mounted(sub)); + + /* Walk a sibling path through the same directory, repeatedly. */ + snprintf(probe, sizeof(probe), "%s/cgroup.procs", self->child); + for (i = 0; i < 8; i++) { + int fd = open(probe, O_RDONLY | O_CLOEXEC); + + if (fd >= 0) + close(fd); + } + + EXPECT_TRUE(path_is_mounted(sub)); + + umount2(sub, MNT_DETACH); + rmdir(sub); + rmdir(src); +} + +/* + * A cached negative dentry must be invalidated when the kernel creates the + * name behind the dcache's back. That is what kernfs_dir_changed() and + * kernfs_elem_dir::rev are for. + */ +TEST_F(kernfs_cgroup, negative_dentry_invalidated_by_kernel_create) +{ + struct stat st; + + /* Caches a negative dentry for the probe file. */ + ASSERT_EQ(stat(self->probe, &st), -1); + ASSERT_EQ(errno, ENOENT); + + /* The kernel now creates it, with no VFS operation on that name. */ + ASSERT_EQ(write_file(self->scratch_sc, self->enable), 0); + + EXPECT_EQ(stat(self->probe, &st), 0); +} + +/* The mirror image: a cached positive dentry must go when the node does. */ +TEST_F(kernfs_cgroup, positive_dentry_invalidated_by_kernel_remove) +{ + struct stat st; + + ASSERT_EQ(write_file(self->scratch_sc, self->enable), 0); + /* Caches a positive dentry. */ + ASSERT_EQ(stat(self->probe, &st), 0); + + ASSERT_EQ(write_file(self->scratch_sc, self->disable), 0); + + ASSERT_EQ(stat(self->probe, &st), -1); + EXPECT_EQ(errno, ENOENT); +} + +/* Opening a removed node fails; it never returns stale content. */ +TEST_F(kernfs_cgroup, open_after_rmdir_fails) +{ + char path[PATH_MAX]; + char buf[64]; + int fd; + + snprintf(path, sizeof(path), "%s/cgroup.procs", self->child); + + fd = open(path, O_RDONLY | O_CLOEXEC); + ASSERT_GE(fd, 0); + + ASSERT_EQ(rmdir(self->child), 0); + + /* Lookup by path must fail. */ + EXPECT_EQ(open(path, O_RDONLY | O_CLOEXEC), -1); + EXPECT_EQ(errno, ENOENT); + + /* + * An fd held across removal must fail rather than return stale + * content. rmdir() deactivates the node before it returns, so + * kernfs_seq_start() fails to get an active reference. + */ + EXPECT_LT(read(fd, buf, sizeof(buf)), 0); + EXPECT_EQ(errno, ENODEV); + EXPECT_EQ(close(fd), 0); + + ASSERT_EQ(mkdir(self->child, 0755), 0); +} + +/* readdir returns every entry exactly once. */ +TEST_F(kernfs_cgroup, readdir_no_duplicates) +{ + char names[512][NAME_MAX + 1]; + struct dirent *de; + int n = 0, i, j; + DIR *d; + + ASSERT_EQ(write_file(self->scratch_sc, self->enable), 0); + + d = opendir(self->child); + ASSERT_NE(d, NULL); + while ((de = readdir(d))) { + if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) + continue; + ASSERT_LT(n, (int)ARRAY_SIZE(names)); + strncpy(names[n], de->d_name, NAME_MAX); + names[n][NAME_MAX] = '\0'; + n++; + } + closedir(d); + + ASSERT_GT(n, 0); + for (i = 0; i < n; i++) + for (j = i + 1; j < n; j++) + EXPECT_STRNE(names[i], names[j]); +} + +#define RESUME_DIRS 24 + +/* + * Resuming at an entry that has gone must carry on after it, never before. + * Take a cookie for every entry, then remove each one, seek to its cookie + * and read the rest; nothing already reported may come back. + */ +TEST_F(kernfs_cgroup, readdir_resume_at_removed_entry) +{ + /* The cgroup's own control files are listed alongside ours. */ + char names[128][NAME_MAX + 1]; + long pos[128]; + char path[PATH_MAX]; + struct dirent *de; + int n = 0, i, j; + DIR *d; + + for (i = 0; i < RESUME_DIRS; i++) { + snprintf(path, sizeof(path), "%s/e%02d", self->scratch, i); + ASSERT_EQ(mkdir(path, 0755), 0); + } + + /* Record the cookie before reading each entry, with its name. */ + d = opendir(self->scratch); + ASSERT_NE(d, NULL); + while (1) { + long here = telldir(d); + + de = readdir(d); + if (!de) + break; + if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) + continue; + ASSERT_LT(n, (int)ARRAY_SIZE(pos)); + pos[n] = here; + strncpy(names[n], de->d_name, NAME_MAX); + names[n][NAME_MAX] = '\0'; + n++; + } + closedir(d); + ASSERT_GT(n, 1); + + for (i = 0; i < n; i++) { + /* Only the directories we made can be removed and put back. */ + if (strncmp(names[i], "e", 1)) + continue; + + snprintf(path, sizeof(path), "%s/%s", self->scratch, names[i]); + ASSERT_EQ(rmdir(path), 0); + + /* Reopen so the seek has to reach the kernel. */ + d = opendir(self->scratch); + ASSERT_NE(d, NULL); + seekdir(d, pos[i]); + while ((de = readdir(d))) { + if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) + continue; + for (j = 0; j < i; j++) + ASSERT_STRNE(de->d_name, names[j]) + TH_LOG("resuming at %s (gone) went back to %s", + names[i], names[j]); + } + closedir(d); + + ASSERT_EQ(mkdir(path, 0755), 0); + } + + for (i = 0; i < RESUME_DIRS; i++) { + snprintf(path, sizeof(path), "%s/e%02d", self->scratch, i); + EXPECT_EQ(rmdir(path), 0); + } +} + +#define CHURN_ROUNDS 400 +#define CHURN_BUFSZ 512 /* small, so a listing takes several calls */ + +/* + * The files appear at the end of the enabling write and go at the start of + * the disabling one, so the window where they exist is the short one. + */ +#define CHURN_DWELL_ON 2000 +#define CHURN_DWELL_OFF 200 + +struct kernfs_dirent64 { + unsigned long long d_ino; + long long d_off; + unsigned short d_reclen; + unsigned char d_type; + char d_name[]; +}; + +/* + * The same resume, but inside one getdents(2) call. rmdir(2) cannot reach + * that window because iterate_dir() holds the listed directory's i_rwsem + * for the whole listing; cgroup.subtree_control can, having no VFS + * operation on the names it adds and removes. The files that are not the + * controller's stay throughout, so each must appear exactly once. + * + * A stress test: it has not been seen to catch the ordering bug, and is + * here to keep the unlocked window under load for lockdep and KASAN. + */ +TEST_F(kernfs_cgroup, readdir_resume_vs_internal_remove) +{ + char buf[CHURN_BUFSZ] __attribute__((aligned(8))); + char stable[128][NAME_MAX + 1]; + int nstable = 0, i, r; + int withctl = 0, without = 0; + int seen[128], status; + pid_t churner; + DIR *d; + + /* With the controller off, whatever is left is what must persist. */ + ASSERT_EQ(write_file(self->scratch_sc, self->disable), 0); + d = opendir(self->child); + ASSERT_NE(d, NULL); + for (;;) { + struct dirent *de = readdir(d); + + if (!de) + break; + if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) + continue; + ASSERT_LT(nstable, (int)ARRAY_SIZE(stable)); + strncpy(stable[nstable], de->d_name, NAME_MAX); + stable[nstable][NAME_MAX] = '\0'; + nstable++; + } + closedir(d); + ASSERT_GT(nstable, 0); + + churner = fork(); + ASSERT_GE(churner, 0); + if (churner == 0) { + for (;;) { + if (write_file(self->scratch_sc, self->enable)) + _exit(10); + usleep(CHURN_DWELL_ON); + if (write_file(self->scratch_sc, self->disable)) + _exit(11); + usleep(CHURN_DWELL_OFF); + } + } + + for (r = 0; r < CHURN_ROUNDS; r++) { + int fd = open(self->child, O_RDONLY | O_DIRECTORY); + int extra = 0; + int n; + ASSERT_GE(fd, 0); + memset(seen, 0, sizeof(seen)); + + while ((n = syscall(SYS_getdents64, fd, buf, sizeof(buf))) > 0) { + int off = 0; + + while (off < n) { + struct kernfs_dirent64 *de = (void *)(buf + off); + bool known = false; + + off += de->d_reclen; + for (i = 0; i < nstable; i++) + if (!strcmp(de->d_name, stable[i])) { + seen[i]++; + known = true; + } + if (!known && strcmp(de->d_name, ".") && + strcmp(de->d_name, "..")) + extra++; + } + } + ASSERT_GE(n, 0); + EXPECT_EQ(close(fd), 0); + + if (extra) + withctl++; + else + without++; + + for (i = 0; i < nstable; i++) + ASSERT_EQ(seen[i], 1) + TH_LOG("round %d: %s seen %d times", + r, stable[i], seen[i]); + } + + /* The churn must have been running, or the listings prove nothing. */ + EXPECT_EQ(kill(churner, SIGKILL), 0); + ASSERT_EQ(waitpid(churner, &status, 0), churner); + ASSERT_TRUE(WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) + TH_LOG("churner exited on its own: status %d", status); + + /* + * They also have to have overlapped it. How much depends on the + * machine, so say the race could not be arranged rather than fail. + */ + if (!withctl || !without) + SKIP(return, "listings did not span the churn: %d with, %d without", + withctl, without); +} + +/* + * A telldir() cookie must resolve back to the same entry after seekdir(). + * kernfs encodes the cookie as the node's name hash, so this covers + * kernfs_dir_pos() as well as plain iteration. + */ +TEST_F(kernfs_cgroup, readdir_seekdir_roundtrip) +{ + char names[512][NAME_MAX + 1]; + struct dirent *de; + long pos[512]; + int n = 0, i; + DIR *d; + + ASSERT_EQ(write_file(self->scratch_sc, self->enable), 0); + + d = opendir(self->child); + ASSERT_NE(d, NULL); + + /* Record the cookie *before* reading each entry, with its name. */ + while (1) { + long here = telldir(d); + + de = readdir(d); + if (!de) + break; + if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) + continue; + ASSERT_LT(n, (int)ARRAY_SIZE(pos)); + pos[n] = here; + strncpy(names[n], de->d_name, NAME_MAX); + names[n][NAME_MAX] = '\0'; + n++; + } + ASSERT_GT(n, 0); + + /* Seeking back to a cookie must land on the entry it was taken at. */ + for (i = 0; i < n; i++) { + seekdir(d, pos[i]); + de = readdir(d); + ASSERT_NE(de, NULL); + EXPECT_STREQ(de->d_name, names[i]); + } + + closedir(d); +} + +#define STRESS_SECS 2 +#define STRESS_DIRS 4 +#define STRESS_READERS 4 + +/* + * Hammer lookup against creation and removal. Revalidation holds no lock + * against the writers, so what makes it safe is that every answer it can + * give is one the caller already handles: a reader must only ever see + * success or an errno meaning "it went away", never garbage or a hang. + */ +TEST_F(kernfs_cgroup, lookup_vs_create_remove_stress) +{ + pid_t pids[STRESS_DIRS + STRESS_READERS]; + struct timespec end; + int i, status, n = 0; + + clock_gettime(CLOCK_MONOTONIC, &end); + end.tv_sec += STRESS_SECS; + + for (i = 0; i < STRESS_DIRS; i++) { + pid_t pid = fork(); + + ASSERT_GE(pid, 0); + if (pid == 0) { + char dir[PATH_MAX]; + + snprintf(dir, sizeof(dir), "%s/s%d", self->scratch, i); + while (!stress_deadline(&end)) { + if (mkdir(dir, 0755) && errno != EEXIST) + _exit(10); + if (rmdir(dir) && errno != ENOENT && + errno != EBUSY) + _exit(11); + } + _exit(0); + } + pids[n++] = pid; + } + + for (i = 0; i < STRESS_READERS; i++) { + pid_t pid = fork(); + + ASSERT_GE(pid, 0); + if (pid == 0) { + /* Start each reader on a different directory. */ + unsigned int seq = i; + + while (!stress_deadline(&end)) { + int which = seq++ % STRESS_DIRS; + char path[PATH_MAX]; + struct stat st; + int fd; + + snprintf(path, sizeof(path), + "%s/s%d/cgroup.procs", + self->scratch, which); + + if (stat(path, &st) && errno != ENOENT && + errno != ENODEV) + _exit(20); + + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) { + if (errno != ENOENT && errno != ENODEV) + _exit(21); + } else { + close(fd); + } + + if (access(path, F_OK) && errno != ENOENT && + errno != ENODEV) + _exit(22); + } + _exit(0); + } + pids[n++] = pid; + } + + for (i = 0; i < n; i++) { + ASSERT_EQ(waitpid(pids[i], &status, 0), pids[i]); + ASSERT_TRUE(WIFEXITED(status)); + EXPECT_EQ(WEXITSTATUS(status), 0); + } + + for (i = 0; i < STRESS_DIRS; i++) { + char dir[PATH_MAX]; + + snprintf(dir, sizeof(dir), "%s/s%d", self->scratch, i); + rmdir_retry(dir); + } +} + +struct kernfs_handle { + struct file_handle h; + unsigned char buf[MAX_HANDLE_SZ]; +}; + +static int kernfs_encode(const char *path, struct kernfs_handle *fh) +{ + int mount_id; + + memset(fh, 0, sizeof(*fh)); + fh->h.handle_bytes = sizeof(fh->buf); + return name_to_handle_at(AT_FDCWD, path, &fh->h, &mount_id, 0); +} + +/* + * Skip only where file handles do not work at all. ENOENT must still + * fail: mkdir leaves a negative dentry cached, so the name resolves only + * after ->d_revalidate() drops it. The encode tests revalidation too. + */ +static bool fh_unsupported(int err) +{ + return err == EOPNOTSUPP || err == EPERM || err == ENOSYS; +} + +/* + * Decoding a file needs CAP_DAC_READ_SEARCH in the initial user + * namespace. Probe once so the tests skip instead of fail. + */ +static bool fh_can_decode(int mfd, struct kernfs_handle *fh) +{ + int fd = open_by_handle_at(mfd, &fh->h, O_PATH); + + if (fd < 0) + return errno != EPERM; + close(fd); + return true; +} + +/* + * A file handle reaches a node without a lookup through its parent. A + * live node must decode. A removed one must not, because + * kernfs_find_and_get_node_by_id() refuses inactive nodes. + * + * Use O_PATH: opening a removed node fails with ENODEV, which would hide + * what is being tested. + */ +TEST_F(kernfs_cgroup, exportfs_decode_and_stale) +{ + char victim[PATH_MAX], procs[PATH_MAX]; + struct kernfs_handle fh; + struct stat st; + int mfd, fd; + + snprintf(victim, sizeof(victim), "%s/fh", self->scratch); + snprintf(procs, sizeof(procs), "%s/cgroup.procs", victim); + ASSERT_EQ(mkdir(victim, 0755), 0); + + /* Any fd on the filesystem identifies it to open_by_handle_at(). */ + mfd = open(self->scratch, O_RDONLY | O_DIRECTORY | O_CLOEXEC); + ASSERT_GE(mfd, 0); + + if (kernfs_encode(procs, &fh)) { + int err = errno; + + close(mfd); + rmdir(victim); + ASSERT_TRUE(fh_unsupported(err)) + TH_LOG("name_to_handle_at: %s", strerror(err)); + SKIP(return, "name_to_handle_at: %s", strerror(err)); + } + + if (!fh_can_decode(mfd, &fh)) { + close(mfd); + rmdir(victim); + SKIP(return, "open_by_handle_at: no CAP_DAC_READ_SEARCH"); + } + + fd = open_by_handle_at(mfd, &fh.h, O_PATH); + ASSERT_GE(fd, 0); + EXPECT_EQ(fstat(fd, &st), 0); + EXPECT_EQ(st.st_nlink, 1); + EXPECT_EQ(close(fd), 0); + + ASSERT_EQ(rmdir(victim), 0); + + fd = open_by_handle_at(mfd, &fh.h, O_PATH); + EXPECT_LT(fd, 0); + if (fd >= 0) + close(fd); + else + EXPECT_EQ(errno, ESTALE); + + EXPECT_EQ(close(mfd), 0); +} + +#define FH_STRESS_SECS 2 +#define FH_DECODE_CAP 10000 + +/* + * Decode file handles while the node is being removed. A decode must + * answer with a usable handle or ESTALE, never garbage and never a hang. + * + * The link count is checked too. An inode that reaches the inode hash + * after the removal cleared link counts keeps the 1 it was born with, so + * it never gets an IN_DELETE_SELF. This has not been seen to fire: it + * needs the decode to stall between the lookup by id and the hash insert, + * and nothing there blocks. It is kept because it is cheap and only + * looks once the directory is gone, so it cannot fail falsely. + */ +TEST_F(kernfs_cgroup, exportfs_decode_vs_rmdir_stress) +{ + int mfd, bad = 0, rounds = 0; + struct timespec end; + + mfd = open(self->scratch, O_RDONLY | O_DIRECTORY | O_CLOEXEC); + ASSERT_GE(mfd, 0); + + clock_gettime(CLOCK_MONOTONIC, &end); + end.tv_sec += FH_STRESS_SECS; + + while (!stress_deadline(&end)) { + char victim[PATH_MAX], procs[PATH_MAX]; + int last = -1, fd, i; + struct kernfs_handle fh; + struct stat st; + pid_t pid; + + snprintf(victim, sizeof(victim), "%s/fh%d", self->scratch, + rounds++); + snprintf(procs, sizeof(procs), "%s/cgroup.procs", victim); + if (mkdir(victim, 0755)) + break; + if (kernfs_encode(procs, &fh)) { + int err = errno; + + rmdir(victim); + ASSERT_TRUE(fh_unsupported(err)) + TH_LOG("name_to_handle_at: %s", strerror(err)); + SKIP(goto out, "name_to_handle_at: %s", strerror(err)); + } + if (rounds == 1 && !fh_can_decode(mfd, &fh)) { + rmdir(victim); + SKIP(goto out, + "open_by_handle_at: no CAP_DAC_READ_SEARCH"); + } + + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) { + rmdir_retry(victim); + _exit(0); + } + + /* + * Decode until the removal deactivates the node. Keep the + * last one that worked: it ran closest to the removal. + */ + for (i = 0; i < FH_DECODE_CAP; i++) { + fd = open_by_handle_at(mfd, &fh.h, O_PATH); + if (fd < 0) + break; + if (last >= 0) + close(last); + last = fd; + } + ASSERT_EQ(waitpid(pid, NULL, 0), pid); + + if (last >= 0) { + if (access(victim, F_OK) && errno == ENOENT && + !fstat(last, &st) && st.st_nlink != 0) + bad++; + close(last); + } + rmdir(victim); + } + + EXPECT_EQ(bad, 0) + TH_LOG("%d of %d rounds decoded a removed node whose inode kept its link count", + bad, rounds); +out: + close(mfd); +} + +/* + * sysfs is namespace tagged (KERNFS_NS) and supports rename; cgroup2 does + * neither. Run in a private netns with its own sysfs so the host is + * untouched. + */ +FIXTURE(kernfs_netns) +{ + char mnt[PATH_MAX]; + char net[PATH_MAX]; + bool mounted; +}; + +FIXTURE_SETUP(kernfs_netns) +{ + if (geteuid()) + SKIP(return, "test needs to run as root"); + + if (unshare(CLONE_NEWNS | CLONE_NEWNET)) + SKIP(return, "unshare(CLONE_NEWNS|CLONE_NEWNET): %s", + strerror(errno)); + + /* Don't let our sysfs mount escape into the parent namespace. */ + ASSERT_EQ(mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL), 0); + + strcpy(self->mnt, "/tmp/kernfs_selftest_sysfs.XXXXXX"); + if (!mkdtemp(self->mnt)) + SKIP(return, "mkdtemp: %s", strerror(errno)); + + if (mount("none", self->mnt, "sysfs", 0, NULL)) { + rmdir(self->mnt); + SKIP(return, "mount sysfs: %s", strerror(errno)); + } + self->mounted = true; + + snprintf(self->net, sizeof(self->net), "%s/class/net", self->mnt); +} + +FIXTURE_TEARDOWN(kernfs_netns) +{ + if (self->mounted) + umount2(self->mnt, MNT_DETACH); + rmdir(self->mnt); +} + +/* + * sysfs must show this network namespace's interfaces, not the parent's. + * + * Do not assume a fresh netns contains only "lo": fallback tunnel devices + * (tunl0, sit0, gre0, ...) are created in every namespace unless + * net.core.fb_tunnels_only_for_init_net is set, so which names appear + * depends on the modules the host has. Check the set instead -- + * if_nametoindex() resolves in the current netns, so every name sysfs shows + * must resolve there, and the counts must agree. + * + * Count only symlinks. Not every entry is a device: bonding adds a + * bonding_masters attribute to /sys/class/net in every namespace. + */ +TEST_F(kernfs_netns, ns_tag_isolates_class_net) +{ + struct if_nameindex *idx, *i; + bool found_lo = false; + int n = 0, want = 0; + struct dirent *de; + DIR *d; + + d = opendir(self->net); + ASSERT_NE(d, NULL); + while ((de = readdir(d))) { + if (de->d_type != DT_LNK) + continue; + EXPECT_NE(if_nametoindex(de->d_name), 0u) + TH_LOG("%s is not in this netns", de->d_name); + if (!strcmp(de->d_name, "lo")) + found_lo = true; + n++; + } + closedir(d); + + idx = if_nameindex(); + ASSERT_NE(idx, NULL); + for (i = idx; i->if_index; i++) + want++; + if_freenameindex(idx); + + EXPECT_TRUE(found_lo); + EXPECT_EQ(n, want); +} + +/* + * After a rename the old name must stop resolving and the new one must + * start, even though both dentries are already cached. + */ +TEST_F(kernfs_netns, rename_is_revalidated) +{ + char old_path[PATH_MAX], new_path[PATH_MAX]; + struct ifreq ifr = {}; + struct stat st; + int sk; + + snprintf(old_path, sizeof(old_path), "%s/lo", self->net); + snprintf(new_path, sizeof(new_path), "%s/%s", self->net, TEST_IFNAME); + + /* Warm both dentries: one positive, one negative. */ + ASSERT_EQ(stat(old_path, &st), 0); + ASSERT_EQ(stat(new_path, &st), -1); + ASSERT_EQ(errno, ENOENT); + + sk = socket(AF_INET, SOCK_DGRAM | SOCK_CLOEXEC, 0); + ASSERT_GE(sk, 0); + strcpy(ifr.ifr_name, "lo"); + strcpy(ifr.ifr_newname, TEST_IFNAME); + if (ioctl(sk, SIOCSIFNAME, &ifr)) { + close(sk); + SKIP(return, "SIOCSIFNAME: %s", strerror(errno)); + } + close(sk); + + EXPECT_EQ(stat(old_path, &st), -1); + EXPECT_EQ(errno, ENOENT); + EXPECT_EQ(stat(new_path, &st), 0); +} + +static int netdev_rename(const char *from, const char *to) +{ + struct ifreq ifr = {}; + int sk, ret; + + sk = socket(AF_INET, SOCK_DGRAM | SOCK_CLOEXEC, 0); + if (sk < 0) + return -1; + strncpy(ifr.ifr_name, from, IFNAMSIZ - 1); + strncpy(ifr.ifr_newname, to, IFNAMSIZ - 1); + ret = ioctl(sk, SIOCSIFNAME, &ifr); + close(sk); + return ret; +} + +/* + * Bounded by a count, not by time: every rename is logged and not rate + * limited, so a timed loop would flood the kernel log. + */ +#define RENAME_FLIPS 200 +#define RENAME_READERS 4 + +/* + * Rename an interface while other tasks look up the names it moves + * between. This renames its /sys/class/net entry through + * kernfs_rename_ns() with the parent unchanged. + * + * The renamer checks what is certain: SIOCSIFNAME returns once the rename + * is done and nothing else renames here, so the new name must resolve and + * the old must not. The readers cannot check that, because the name can + * move between their two lstat() calls. They only check that a lookup + * returns success or ENOENT, and keep the lock busy while renames run. + * + * lstat() not stat(): /sys/class/net/ is a symlink and is renamed + * before the directory it points at, so the two are not atomic. + */ +TEST_F(kernfs_netns, rename_vs_lookup_stress) +{ + char old_path[PATH_MAX], new_path[PATH_MAX]; + pid_t pids[RENAME_READERS]; + int i, status, n = 0, bad = 0; + struct stat st; + int done[2]; + + snprintf(old_path, sizeof(old_path), "%s/lo", self->net); + snprintf(new_path, sizeof(new_path), "%s/%s", self->net, TEST_IFNAME); + + if (netdev_rename("lo", TEST_IFNAME)) + SKIP(return, "SIOCSIFNAME: %s", strerror(errno)); + if (netdev_rename(TEST_IFNAME, "lo")) + SKIP(return, "SIOCSIFNAME back: %s", strerror(errno)); + + /* Readers run until the renamer closes the write end. */ + ASSERT_EQ(pipe2(done, O_NONBLOCK | O_CLOEXEC), 0); + + for (i = 0; i < RENAME_READERS; i++) { + pid_t pid = fork(); + + ASSERT_GE(pid, 0); + if (pid == 0) { + struct stat rst; + char c; + + close(done[1]); + while (read(done[0], &c, 1) < 0 && errno == EAGAIN) { + if (lstat(old_path, &rst) && errno != ENOENT) + _exit(20); + if (lstat(new_path, &rst) && errno != ENOENT) + _exit(21); + } + _exit(0); + } + pids[n++] = pid; + } + close(done[0]); + + for (i = 0; i < RENAME_FLIPS; i++) { + if (netdev_rename("lo", TEST_IFNAME)) + break; + if (lstat(new_path, &st) || !lstat(old_path, &st)) { + bad++; + break; + } + if (netdev_rename(TEST_IFNAME, "lo")) + break; + if (lstat(old_path, &st) || !lstat(new_path, &st)) { + bad++; + break; + } + } + close(done[1]); + + for (i = 0; i < n; i++) { + ASSERT_EQ(waitpid(pids[i], &status, 0), pids[i]); + ASSERT_TRUE(WIFEXITED(status)); + EXPECT_EQ(WEXITSTATUS(status), 0); + } + + EXPECT_EQ(bad, 0) + TH_LOG("a completed rename left the wrong name resolving"); + + /* Leave the interface as the fixture found it. */ + netdev_rename(TEST_IFNAME, "lo"); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/filesystems/open_tree_ns/open_tree_ns_test.c b/tools/testing/selftests/filesystems/open_tree_ns/open_tree_ns_test.c index 82f3c8c02c9ac8..438054ab6a6a4c 100644 --- a/tools/testing/selftests/filesystems/open_tree_ns/open_tree_ns_test.c +++ b/tools/testing/selftests/filesystems/open_tree_ns/open_tree_ns_test.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -32,6 +33,10 @@ #define OPEN_TREE_NAMESPACE (1 << 1) #endif +#ifndef OPEN_TREE_DROP_MNTNS_MOUNTS +#define OPEN_TREE_DROP_MNTNS_MOUNTS (1 << 2) +#endif + static int get_mnt_ns_id(int fd, uint64_t *mnt_ns_id) { if (ioctl(fd, NS_GET_MNTNS_ID, mnt_ns_id) < 0) @@ -1004,4 +1009,192 @@ TEST_F(open_tree_ns_unbindable, recursive_skips_on_unbindable) close(fd); } +/* + * Pin a mount namespace at @where, the way snapd does under /run/snapd/ns. + * The namespace has to be younger than ours, so a child unshares and the + * parent binds the child's namespace file: binding one's own namespace is + * refused by the very check this test is about. + */ +static int pin_mount_namespace(const char *where) +{ + int pipefd[2], status, ret = -1; + char path[PATH_MAX]; + pid_t pid; + char c; + + if (pipe(pipefd)) + return -1; + + pid = fork(); + if (pid < 0) + goto out; + if (pid == 0) { + close(pipefd[0]); + if (unshare(CLONE_NEWNS)) + _exit(1); + /* Tell the parent the namespace exists, then hold it open. */ + if (write(pipefd[1], "x", 1) != 1) + _exit(1); + pause(); + _exit(0); + } + + close(pipefd[1]); + pipefd[1] = -1; + if (read(pipefd[0], &c, 1) != 1) + goto out_kill; + + snprintf(path, sizeof(path), "/proc/%d/ns/mnt", pid); + if (mount(path, where, NULL, MS_BIND, NULL)) + goto out_kill; + + ret = 0; + +out_kill: + kill(pid, SIGKILL); + waitpid(pid, &status, 0); +out: + close(pipefd[0]); + if (pipefd[1] >= 0) + close(pipefd[1]); + return ret; +} + +FIXTURE(open_tree_ns_drop_mntns) +{ + char dir[64]; + char pin[PATH_MAX]; + bool mounted; +}; + +FIXTURE_SETUP(open_tree_ns_drop_mntns) +{ + int fd, ret; + + self->mounted = false; + snprintf(self->dir, sizeof(self->dir), "/tmp/open_tree_ns_drop_mntns.XXXXXX"); + + ret = sys_open_tree(-1, NULL, 0); + if (ret == -1 && errno == ENOSYS) + SKIP(return, "open_tree() syscall not supported"); + + /* + * Work in a private mount namespace, so whatever is mounted here, + * the pinned namespace included, goes away with the test process. + */ + if (unshare(CLONE_NEWNS)) + SKIP(return, "unshare(CLONE_NEWNS) failed: %s", strerror(errno)); + ASSERT_EQ(mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL), 0); + + ASSERT_NE(mkdtemp(self->dir), NULL); + if (mount("tmpfs", self->dir, "tmpfs", 0, NULL)) + SKIP(return, "Failed to mount tmpfs"); + self->mounted = true; + + fd = sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_CLONE | OPEN_TREE_DROP_MNTNS_MOUNTS | + AT_RECURSIVE | OPEN_TREE_CLOEXEC); + if (fd < 0 && errno == EINVAL) + SKIP(return, "OPEN_TREE_DROP_MNTNS_MOUNTS not supported"); + ASSERT_GE(fd, 0); + close(fd); + + snprintf(self->pin, sizeof(self->pin), "%s/ns", self->dir); + fd = open(self->pin, O_CREAT | O_RDONLY | O_CLOEXEC, 0600); + ASSERT_GE(fd, 0); + close(fd); + ASSERT_EQ(pin_mount_namespace(self->pin), 0); +} + +FIXTURE_TEARDOWN(open_tree_ns_drop_mntns) +{ + if (self->mounted) + umount2(self->dir, MNT_DETACH); + rmdir(self->dir); +} + +/* Attach @clone_fd at "/" inside the namespace @ns_fd; returns errno. */ +static int move_into_namespace(int clone_fd, int ns_fd) +{ + pid_t pid; + int status; + + pid = fork(); + if (pid < 0) + return -1; + if (pid == 0) { + int target; + + if (setns(ns_fd, CLONE_NEWNS)) + _exit(255); + target = open("/", O_PATH | O_DIRECTORY | O_CLOEXEC); + if (target < 0) + _exit(255); + if (sys_move_mount(clone_fd, "", target, "", + MOVE_MOUNT_F_EMPTY_PATH | MOVE_MOUNT_T_EMPTY_PATH)) + _exit(errno); + _exit(0); + } + if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status)) + return -1; + return WEXITSTATUS(status); +} + +TEST_F(open_tree_ns_drop_mntns, move_into_younger_namespace) +{ + int clone_fd, ns_fd; + + ns_fd = sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_NAMESPACE | OPEN_TREE_CLOEXEC); + ASSERT_GE(ns_fd, 0); + + /* + * Without the flag the clone carries the pinned namespace along, + * and move_mount() refuses it from inside a younger namespace. + */ + clone_fd = sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_CLONE | AT_RECURSIVE | OPEN_TREE_CLOEXEC); + ASSERT_GE(clone_fd, 0); + EXPECT_EQ(move_into_namespace(clone_fd, ns_fd), ELOOP); + close(clone_fd); + + /* With the flag the pinned namespace is left out. */ + clone_fd = sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_CLONE | OPEN_TREE_DROP_MNTNS_MOUNTS | + AT_RECURSIVE | OPEN_TREE_CLOEXEC); + ASSERT_GE(clone_fd, 0); + EXPECT_EQ(move_into_namespace(clone_fd, ns_fd), 0); + close(clone_fd); + + close(ns_fd); +} + +TEST_F(open_tree_ns_drop_mntns, flag_combinations) +{ + int fd; + + /* Nothing is copied, so there is nothing to drop. */ + EXPECT_LT(sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_DROP_MNTNS_MOUNTS | OPEN_TREE_CLOEXEC), 0); + EXPECT_EQ(errno, EINVAL); + + /* A non-recursive clone has no submounts, so there is nothing to drop. */ + EXPECT_LT(sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_CLONE | OPEN_TREE_DROP_MNTNS_MOUNTS | + OPEN_TREE_CLOEXEC), 0); + EXPECT_EQ(errno, EINVAL); + EXPECT_LT(sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_NAMESPACE | OPEN_TREE_DROP_MNTNS_MOUNTS | + OPEN_TREE_CLOEXEC), 0); + EXPECT_EQ(errno, EINVAL); + + /* OPEN_TREE_NAMESPACE never copies them; the flag is a no-op there. */ + fd = sys_open_tree(AT_FDCWD, self->dir, + OPEN_TREE_NAMESPACE | OPEN_TREE_DROP_MNTNS_MOUNTS | + AT_RECURSIVE | OPEN_TREE_CLOEXEC); + EXPECT_GE(fd, 0); + if (fd >= 0) + close(fd); +} + TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/filesystems/openat2/openat2_test.c b/tools/testing/selftests/filesystems/openat2/openat2_test.c index 6f5afbe2d8d310..e08c94ce05305e 100644 --- a/tools/testing/selftests/filesystems/openat2/openat2_test.c +++ b/tools/testing/selftests/filesystems/openat2/openat2_test.c @@ -23,8 +23,16 @@ * XXX: This is wrong on {mips, parisc, powerpc, sparc}. */ #undef O_LARGEFILE -#ifdef __aarch64__ +#if defined(__aarch64__) || defined(__alpha__) #define O_LARGEFILE 0x20000 +#elif defined(__powerpc__) || defined(__ppc__) +#define O_LARGEFILE 0x10000 +#elif defined(__sparc__) +#define O_LARGEFILE 0x40000 +#elif defined(__mips__) +#define O_LARGEFILE 0x2000 +#elif defined(__parisc__) +#define O_LARGEFILE 0x800 #else #define O_LARGEFILE 0x8000 #endif diff --git a/tools/testing/selftests/filesystems/openat2/resolve_test.c b/tools/testing/selftests/filesystems/openat2/resolve_test.c index eacde59ce15830..6216a854638847 100644 --- a/tools/testing/selftests/filesystems/openat2/resolve_test.c +++ b/tools/testing/selftests/filesystems/openat2/resolve_test.c @@ -140,9 +140,12 @@ FIXTURE_SETUP(openat2_resolve) if (!openat2_supported) SKIP(return, "openat2(2) not supported"); - /* Unshare and make /tmp a new directory. */ + /* Unshare and make the mount tree private. */ ASSERT_EQ(unshare(CLONE_NEWNS), 0); - ASSERT_EQ(mount("", "/tmp", "", MS_PRIVATE, ""), 0); + ASSERT_EQ(mount("", "/", "", MS_PRIVATE | MS_REC, ""), 0); + + /* Ensure /tmp is a mountpoint for RESOLVE_NO_XDEV test crossing into /tmp. */ + ASSERT_EQ(mount("/tmp", "/tmp", NULL, MS_BIND, NULL), 0); /* Make the top-level directory. */ ASSERT_NE(mkdtemp(dirname), NULL); diff --git a/tools/testing/selftests/filesystems/statmount/statmount_test.c b/tools/testing/selftests/filesystems/statmount/statmount_test.c index 60c2c544db6a48..52544512edc757 100644 --- a/tools/testing/selftests/filesystems/statmount/statmount_test.c +++ b/tools/testing/selftests/filesystems/statmount/statmount_test.c @@ -17,7 +17,7 @@ static const char *const known_fs[] = { "9p", "adfs", "affs", "afs", "aio", "anon_inodefs", "apparmorfs", - "autofs", "bcachefs", "bdev", "befs", "bfs", "binder", "binfmt_misc", + "autofs", "bcachefs", "bdev", "befs", "binder", "binfmt_misc", "bpf", "btrfs", "btrfs_test_fs", "ceph", "cgroup", "cgroup2", "cifs", "coda", "configfs", "cpuset", "cramfs", "cxl", "dax", "debugfs", "devpts", "devtmpfs", "dmabuf", "drm", "ecryptfs", "efivarfs", "efs", diff --git a/tools/testing/selftests/kho/vmtest.sh b/tools/testing/selftests/kho/vmtest.sh index 0014bd76e88de6..1e86ffcc3a79e6 100755 --- a/tools/testing/selftests/kho/vmtest.sh +++ b/tools/testing/selftests/kho/vmtest.sh @@ -100,7 +100,7 @@ file /init $tmp_dir/init 0755 0 0 file /kernel $kernel 0644 0 0 EOF - "$build_dir/usr/gen_init_cpio" "$tmp_dir/cpio_list" > "$initrd" + "$build_dir/scripts/gen_init_cpio" "$tmp_dir/cpio_list" > "$initrd" } function run_qemu() { diff --git a/tools/testing/selftests/landlock/fs_test.c b/tools/testing/selftests/landlock/fs_test.c index 6e979cef884d15..fd20a2b3c0a5ef 100644 --- a/tools/testing/selftests/landlock/fs_test.c +++ b/tools/testing/selftests/landlock/fs_test.c @@ -4146,6 +4146,9 @@ TEST_F_FORK(layout1, o_path_ftruncate_and_ioctl) ASSERT_EQ(0, close(fd)); } +/* Arbitrary command with nonzero bits in both 16-bit halves. */ +static const unsigned int unknown_ioctl_cmd = 0xc00ffeee; + /* * ioctl_error - generically call the given ioctl with a pointer to a * sufficiently large zeroed-out memory region. @@ -4249,7 +4252,7 @@ TEST_F_FORK(layout1, blanket_permitted_ioctls) EXPECT_EQ(EACCES, ioctl_error(_metadata, fd, FS_IOC_ZERO_RANGE)); /* Default case is also blocked. */ - EXPECT_EQ(EACCES, ioctl_error(_metadata, fd, 0xc00ffeee)); + EXPECT_EQ(EACCES, ioctl_error(_metadata, fd, unknown_ioctl_cmd)); ASSERT_EQ(0, close(fd)); } @@ -7943,6 +7946,7 @@ TEST_F(audit_layout1, truncate) EXPECT_EQ(1, records.domain); } +/* Checks that audit records preserve every ioctl command bit. */ TEST_F(audit_layout1, ioctl_dev) { struct audit_records records; @@ -7952,10 +7956,10 @@ TEST_F(audit_layout1, ioctl_dev) fd = open("/dev/null", O_RDONLY | O_CLOEXEC); ASSERT_LE(0, fd); - EXPECT_EQ(EACCES, ioctl_error(_metadata, fd, FIONREAD)); + EXPECT_EQ(EACCES, ioctl_error(_metadata, fd, unknown_ioctl_cmd)); EXPECT_EQ(0, matches_log_fs_extra(_metadata, self->audit_fd, "fs\\.ioctl_dev", "/dev/null", - " ioctlcmd=0x541b")); + " ioctlcmd=0xc00ffeee")); EXPECT_EQ(0, audit_count_records(self->audit_fd, &records)); EXPECT_EQ(0, records.access); diff --git a/tools/testing/selftests/liveupdate/vmtest.sh b/tools/testing/selftests/liveupdate/vmtest.sh index 64fd2ab55b765f..672a9e9d685a55 100755 --- a/tools/testing/selftests/liveupdate/vmtest.sh +++ b/tools/testing/selftests/liveupdate/vmtest.sh @@ -152,7 +152,7 @@ file /test_binary $workspace_dir/test_binary 0755 0 0 EOF # Generate inner_initrd.cpio - "$build_dir/usr/gen_init_cpio" "$workspace_dir/cpio_list_inner" > "$workspace_dir/inner_initrd.cpio" + "$build_dir/scripts/gen_init_cpio" "$workspace_dir/cpio_list_inner" > "$workspace_dir/inner_initrd.cpio" cat > "$workspace_dir/cpio_list" < "$initrd" + "$build_dir/scripts/gen_init_cpio" "$workspace_dir/cpio_list" > "$initrd" } function run_qemu() { diff --git a/tools/testing/selftests/namespaces/.gitignore b/tools/testing/selftests/namespaces/.gitignore index 0989e80da4571f..995b082357fa66 100644 --- a/tools/testing/selftests/namespaces/.gitignore +++ b/tools/testing/selftests/namespaces/.gitignore @@ -10,3 +10,4 @@ cred_change_test stress_test listns_pagination_bug regression_pidfd_setns_test +uid_map_range_test diff --git a/tools/testing/selftests/namespaces/Makefile b/tools/testing/selftests/namespaces/Makefile index fbb821652c1787..9b22d12708e1aa 100644 --- a/tools/testing/selftests/namespaces/Makefile +++ b/tools/testing/selftests/namespaces/Makefile @@ -13,7 +13,8 @@ TEST_GEN_PROGS := nsid_test \ cred_change_test \ stress_test \ listns_pagination_bug \ - regression_pidfd_setns_test + regression_pidfd_setns_test \ + uid_map_range_test include ../lib.mk diff --git a/tools/testing/selftests/namespaces/ns_active_ref_test.c b/tools/testing/selftests/namespaces/ns_active_ref_test.c index 093268f0efaa02..ab5ffe4de5213f 100644 --- a/tools/testing/selftests/namespaces/ns_active_ref_test.c +++ b/tools/testing/selftests/namespaces/ns_active_ref_test.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -20,6 +21,10 @@ #include "../filesystems/utils.h" #include "wrappers.h" +#ifndef PIDFD_THREAD +#define PIDFD_THREAD O_EXCL +#endif + #ifndef FD_NSFS_ROOT #define FD_NSFS_ROOT -10003 /* Root of the nsfs filesystem */ #endif @@ -2120,6 +2125,7 @@ TEST(ns_mixed_types_same_owner) /* Thread test helpers and structures */ struct thread_ns_info { __u64 ns_id; + int pidfd; int pipefd; int syncfd_read; int syncfd_write; @@ -2152,6 +2158,13 @@ static void *thread_create_namespace(void *arg) return NULL; } + /* Open a pidfd so the main thread can later poll it to detect actual thread exit. */ + info->pidfd = syscall(__NR_pidfd_open, gettid(), PIDFD_THREAD); + if (info->pidfd < 0) { + info->exit_code = 6; + return NULL; + } + /* Send namespace ID to main thread */ if (write(info->pipefd, &info->ns_id, sizeof(info->ns_id)) != sizeof(info->ns_id)) { info->exit_code = 4; @@ -2169,6 +2182,20 @@ static void *thread_create_namespace(void *arg) return NULL; } +/* + * pthread_join() returns once exit_mm() wakes the CLEARTID futex, before + * exit_nsproxy_namespaces() drops the namespace reference. A pidfd becomes + * readable only from exit_notify(), after that point. + */ +static int wait_thread_exited(int pidfd) +{ + struct pollfd pfd = { .fd = pidfd, .events = POLLIN }; + int ret = poll(&pfd, 1, -1); + + close(pidfd); + return ret; +} + /* * Test that namespace becomes inactive after thread exits. * This verifies active reference counting works with threads, not just processes. @@ -2234,6 +2261,7 @@ TEST(thread_ns_inactive_after_exit) close(syncpipe[1]); /* Wait for thread to exit */ + ASSERT_EQ(wait_thread_exited(info.pidfd), 1); ASSERT_EQ(pthread_join(thread, NULL), 0); close(pipefd[0]); close(pipefd[1]); @@ -2317,6 +2345,7 @@ TEST(thread_ns_fd_keeps_active) close(syncpipe[1]); /* Wait for thread to exit */ + ASSERT_EQ(wait_thread_exited(info.pidfd), 1); pthread_join(thread, NULL); close(pipefd[0]); close(pipefd[1]); diff --git a/tools/testing/selftests/namespaces/uid_map_range_test.c b/tools/testing/selftests/namespaces/uid_map_range_test.c new file mode 100644 index 00000000000000..1eba1e609205ef --- /dev/null +++ b/tools/testing/selftests/namespaces/uid_map_range_test.c @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include "../kselftest_harness.h" + +/* + * The fields of a uid_map/gid_map line are stored in u32. Values that + * do not fit, and negative values, must be rejected with EINVAL instead + * of being silently truncated, and in-range values must still be + * accepted. + */ + +static int write_file(const char *path, const char *buf) +{ + ssize_t len = strlen(buf); + ssize_t ret; + int fd; + + fd = open(path, O_WRONLY | O_CLOEXEC); + if (fd < 0) + return -errno; + ret = write(fd, buf, len); + if (ret < 0) + ret = -errno; + close(fd); + if (ret < 0) + return ret; + return ret == len ? 0 : -EIO; +} + +static ssize_t read_file(const char *path, char *buf, size_t size) +{ + ssize_t ret; + int fd; + + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return -errno; + ret = read(fd, buf, size - 1); + if (ret < 0) + ret = -errno; + else + buf[ret] = '\0'; + close(fd); + return ret; +} + +/* Enter a fresh user namespace, remembering the outer ids. */ +static int enter_userns(uid_t *uid, gid_t *gid) +{ + *uid = geteuid(); + *gid = getegid(); + if (unshare(CLONE_NEWUSER) < 0) + return -errno; + /* Required before an unprivileged gid_map write. */ + return write_file("/proc/self/setgroups", "deny"); +} + +static void expect_rejected(struct __test_metadata *_metadata, + const char *path, const char *line) +{ + char buf[256]; + + TH_LOG("writing \"%s\" to %s", line, path); + EXPECT_EQ(write_file(path, line), -EINVAL); + /* Nothing may have been installed. */ + EXPECT_EQ(read_file(path, buf, sizeof(buf)), 0) { + TH_LOG("installed: %s", buf); + } +} + +static void expect_accepted(struct __test_metadata *_metadata, + const char *path, const char *line, + unsigned int first, unsigned int lower_first, + unsigned int count) +{ + unsigned int f, l, c; + char buf[256]; + + TH_LOG("writing \"%s\" to %s", line, path); + ASSERT_EQ(write_file(path, line), 0); + + ASSERT_GT(read_file(path, buf, sizeof(buf)), 0); + ASSERT_EQ(sscanf(buf, "%u %u %u", &f, &l, &c), 3); + EXPECT_EQ(f, first); + EXPECT_EQ(l, lower_first); + EXPECT_EQ(c, count); + /* Exactly one extent. */ + EXPECT_EQ(strchr(buf, '\n'), buf + strlen(buf) - 1); +} + +TEST(uid_map_first_above_u32) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "4294967301 %u 1", uid); + expect_rejected(_metadata, "/proc/self/uid_map", line); +} + +TEST(uid_map_lower_first_above_u32) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "0 %llu 1", (1ULL << 32) + uid); + expect_rejected(_metadata, "/proc/self/uid_map", line); +} + +TEST(uid_map_count_above_u32) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "0 %u 4294967297", uid); + expect_rejected(_metadata, "/proc/self/uid_map", line); +} + +TEST(gid_map_first_above_u32) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "4294967301 %u 1", gid); + expect_rejected(_metadata, "/proc/self/gid_map", line); +} + +TEST(uid_map_first_negative) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "-1 %u 1", uid); + expect_rejected(_metadata, "/proc/self/uid_map", line); +} + +TEST(uid_map_lower_first_negative) +{ + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + expect_rejected(_metadata, "/proc/self/uid_map", "0 -1 1"); +} + +TEST(uid_map_count_negative) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "0 %u -1", uid); + expect_rejected(_metadata, "/proc/self/uid_map", line); +} + +TEST(uid_map_in_range) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "5 %u 1", uid); + expect_accepted(_metadata, "/proc/self/uid_map", line, 5, uid, 1); +} + +TEST(uid_map_zero) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "0 %u 1", uid); + expect_accepted(_metadata, "/proc/self/uid_map", line, 0, uid, 1); +} + +/* 4294967295 is (u32)-1, which map_write() refuses as a starting id. */ +TEST(uid_map_largest_first) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "4294967294 %u 1", uid); + expect_accepted(_metadata, "/proc/self/uid_map", line, + 4294967294U, uid, 1); +} + +TEST(uid_map_extra_spaces) +{ + char line[64]; + uid_t uid; + gid_t gid; + int ret; + + ret = enter_userns(&uid, &gid); + if (ret) + SKIP(return, "cannot set up user namespace: %s", strerror(-ret)); + snprintf(line, sizeof(line), "0 %u 1", uid); + expect_accepted(_metadata, "/proc/self/uid_map", line, 0, uid, 1); +} + +/* + * An unprivileged writer may only map a single id, so the largest count + * needs a privileged parent writing the map of a child namespace. With + * first and lower_first 0, the no-wrap rule allows up to 4294967295. + */ +TEST(uid_map_largest_count) +{ + unsigned int first, lower_first, count; + int to_child[2], from_child[2]; + char path[64], buf[256]; + pid_t pid; + char c; + + if (geteuid() != 0) + SKIP(return, "needs root in the initial user namespace"); + ASSERT_GT(read_file("/proc/self/uid_map", buf, sizeof(buf)), 0); + ASSERT_EQ(sscanf(buf, "%u %u %u", &first, &lower_first, &count), 3); + if (first != 0 || lower_first != 0 || count != 4294967295U) + SKIP(return, "not in the initial user namespace"); + + ASSERT_EQ(pipe(to_child), 0); + ASSERT_EQ(pipe(from_child), 0); + pid = fork(); + ASSERT_GE(pid, 0); + if (pid == 0) { + close(to_child[1]); + close(from_child[0]); + c = unshare(CLONE_NEWUSER) ? 'F' : 'K'; + if (write(from_child[1], &c, 1) != 1) + _exit(1); + /* Stay alive until the parent is done with our uid_map. */ + if (read(to_child[0], &c, 1) < 0) + _exit(1); + _exit(0); + } + close(to_child[0]); + close(from_child[1]); + + ASSERT_EQ(read(from_child[0], &c, 1), 1); + ASSERT_EQ(c, 'K'); + snprintf(path, sizeof(path), "/proc/%d/uid_map", pid); + expect_accepted(_metadata, path, "0 0 4294967295", 0, 0, 4294967295U); + + close(to_child[1]); + EXPECT_EQ(waitpid(pid, NULL, 0), pid); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/net/af_unix/Makefile b/tools/testing/selftests/net/af_unix/Makefile index a66f10fb0c231d..45b841758f1bfb 100644 --- a/tools/testing/selftests/net/af_unix/Makefile +++ b/tools/testing/selftests/net/af_unix/Makefile @@ -23,6 +23,8 @@ TEST_GEN_FILES := scm_rights_denial_lsm.bpf.o include ../../lib.mk include ../bpf.mk +$(OUTPUT)/scm_pidfd: CFLAGS += -pthread + $(OUTPUT)/scm_rights_denial_lsm: $(BPFOBJ) $(OUTPUT)/scm_rights_denial_lsm: CFLAGS += -I$(SCRATCH_DIR)/include $(OUTPUT)/scm_rights_denial_lsm: LDLIBS += -lelf -lz diff --git a/tools/testing/selftests/net/af_unix/scm_pidfd.c b/tools/testing/selftests/net/af_unix/scm_pidfd.c index 2c18b92a260353..cbe53fab44fc5a 100644 --- a/tools/testing/selftests/net/af_unix/scm_pidfd.c +++ b/tools/testing/selftests/net/af_unix/scm_pidfd.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -27,6 +28,18 @@ #define SCM_PIDFD 0x04 #endif +#ifndef SCM_PIDFD_THREAD +#define SCM_PIDFD_THREAD 0x05 +#endif + +#ifndef SO_PASSPIDFD_THREAD +#define SO_PASSPIDFD_THREAD 86 +#endif + +#ifndef SO_PEERPIDFD_THREAD +#define SO_PEERPIDFD_THREAD 87 +#endif + #define CHILD_EXIT_CODE_OK 123 static void child_die() @@ -553,4 +566,593 @@ TEST_F(scm_pidfd, test) close(pfd); } +struct thread_ids { + pid_t pid; + pid_t tid; +}; + +#define MAX_WRITERS 2 + +/* Used by writers to signal they've written, so we can sequence multiple writers */ +static int seq_pipe[2]; + +static void *send_ids_thread(void *arg) +{ + int fd = *(int *)arg; + struct thread_ids ids = { + .pid = getpid(), + .tid = gettid(), + }; + char sync; + + if (send(fd, &ids, sizeof(ids), 0) != sizeof(ids)) + return (void *)1; + + /* let the client start the next writer, so the queue order is known */ + if (write(seq_pipe[1], "1", 1) != 1) + return (void *)1; + + /* stay alive until the receiver has looked at our pidfd */ + if (read(fd, &sync, 1) != 1) + return (void *)1; + + return NULL; +} + +static void *send_ids_creds_thread(void *arg) +{ + int fd = *(int *)arg; + struct thread_ids ids = { + .pid = getpid(), + .tid = gettid(), + }; + struct ucred ucred = { + .pid = getpid(), + .uid = getuid(), + .gid = getgid(), + }; + char control[CMSG_SPACE(sizeof(ucred))] = { 0 }; + struct iovec iov; + struct msghdr msg = { 0 }; + struct cmsghdr *cmsg; + char sync; + + iov.iov_base = &ids; + iov.iov_len = sizeof(ids); + + msg.msg_iov = &iov; + msg.msg_iovlen = 1; + msg.msg_control = control; + msg.msg_controllen = sizeof(control); + + cmsg = CMSG_FIRSTHDR(&msg); + cmsg->cmsg_level = SOL_SOCKET; + cmsg->cmsg_type = SCM_CREDENTIALS; + cmsg->cmsg_len = CMSG_LEN(sizeof(ucred)); + memcpy(CMSG_DATA(cmsg), &ucred, sizeof(ucred)); + + if (sendmsg(fd, &msg, 0) != sizeof(ids)) + return (void *)1; + + /* signal we've written our info */ + if (write(seq_pipe[1], "1", 1) != 1) + return (void *)1; + + /* stay alive until the receiver has looked at our pidfd */ + if (read(fd, &sync, 1) != 1) + return (void *)1; + + return NULL; +} + +/* + * Runs @nwriters copies of @sender, each in its own non-leader thread, one + * at a time so the queue order is known, then tells the parent on @ackfd + * that everything is queued. + */ +static void thread_client(int fd, int syncfd, int ackfd, + void *(*sender)(void *), int nwriters) +{ + pthread_t writers[MAX_WRITERS]; + void *ret; + char sync; + int i; + + /* wait until the receiver enabled the options it wants */ + if (read(syncfd, &sync, 1) != 1) + child_die(); + + for (i = 0; i < nwriters; i++) { + if (pthread_create(&writers[i], NULL, sender, &fd)) + child_die(); + + if (read(seq_pipe[0], &sync, 1) != 1) + child_die(); + } + + if (write(ackfd, "1", 1) != 1) + child_die(); + + for (i = 0; i < nwriters; i++) + if (pthread_join(writers[i], &ret) || ret) + child_die(); + + exit(0); +} + +struct pidfd_msg { + ssize_t len; + struct thread_ids ids; + struct pidfd_info tgid_info; + struct pidfd_info thread_info; + int tgid_flags; + int thread_flags; + bool have_tgid; + bool have_thread; +}; + +static int get_pidfd_info(int pidfd, struct pidfd_info *info) +{ + info->mask = PIDFD_INFO_PID; + if (ioctl(pidfd, PIDFD_GET_INFO, info)) { + log_err("ioctl(PIDFD_GET_INFO)"); + return -1; + } + + return 0; +} + +/* One recvmsg(), reporting how many bytes came back and what the pidfd + * cmsgs that came with them say. + */ +static int read_pidfd_msg(int fd, struct pidfd_msg *out) +{ + char control[CMSG_SPACE(sizeof(int)) * 2] = { 0 }; + int tgid_pidfd = -1, thread_pidfd = -1; + struct thread_ids buf[2] = { 0 }; + struct msghdr msg = { 0 }; + struct cmsghdr *cmsg; + struct iovec iov; + + iov.iov_base = buf; + iov.iov_len = sizeof(buf); + msg.msg_iov = &iov; + msg.msg_iovlen = 1; + msg.msg_control = control; + msg.msg_controllen = sizeof(control); + + out->len = recvmsg(fd, &msg, 0); + if (out->len < 0) { + log_err("recvmsg"); + return -1; + } + + if (msg.msg_flags & MSG_CTRUNC) { + log_err("recvmsg: control truncated"); + return -1; + } + + out->ids = buf[0]; + + for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != NULL; + cmsg = CMSG_NXTHDR(&msg, cmsg)) { + if (cmsg->cmsg_level != SOL_SOCKET) + continue; + + if (cmsg->cmsg_type == SCM_PIDFD) + memcpy(&tgid_pidfd, CMSG_DATA(cmsg), sizeof(tgid_pidfd)); + else if (cmsg->cmsg_type == SCM_PIDFD_THREAD) + memcpy(&thread_pidfd, CMSG_DATA(cmsg), sizeof(thread_pidfd)); + } + + out->have_tgid = tgid_pidfd >= 0; + out->have_thread = thread_pidfd >= 0; + + if (out->have_tgid) { + if (get_pidfd_info(tgid_pidfd, &out->tgid_info)) + return -1; + out->tgid_flags = fcntl(tgid_pidfd, F_GETFL); + close(tgid_pidfd); + } + + if (out->have_thread) { + if (get_pidfd_info(thread_pidfd, &out->thread_info)) + return -1; + out->thread_flags = fcntl(thread_pidfd, F_GETFL); + close(thread_pidfd); + } + + return 0; +} + +/* + * Runs @nwriters threads of one child process against a SOCK_STREAM pair, + * all of them queued before the parent reads. SO_PASSPIDFD is set when + * @want_tgid is given, SO_PASSPIDFD_THREAD when @want_thread is. @nread + * reads are returned in @out. + */ +static int pidfd_flow(void *(*sender)(void *), int nwriters, bool want_tgid, + bool want_thread, struct pidfd_msg *out, int nread) +{ + int child_status = 0; + int syncpipe[2]; + int ackpipe[2]; + int sk[2]; + int on = 1; + char sync; + pid_t child; + int i; + + if (nwriters > MAX_WRITERS) + return -1; + + if (socketpair(AF_UNIX, SOCK_STREAM, 0, sk)) + return -1; + if (pipe(syncpipe) || pipe(ackpipe) || pipe(seq_pipe)) + return -1; + + child = fork(); + if (child < 0) + return -1; + + if (child == 0) { + close(sk[0]); + close(syncpipe[1]); + close(ackpipe[0]); + thread_client(sk[1], syncpipe[0], ackpipe[1], sender, nwriters); + } + close(sk[1]); + close(syncpipe[0]); + close(ackpipe[1]); + close(seq_pipe[0]); + close(seq_pipe[1]); + + if (want_tgid && + setsockopt(sk[0], SOL_SOCKET, SO_PASSPIDFD, &on, sizeof(on))) { + log_err("Failed to set SO_PASSPIDFD"); + return -1; + } + + if (want_thread && + setsockopt(sk[0], SOL_SOCKET, SO_PASSPIDFD_THREAD, &on, sizeof(on))) { + log_err("Failed to set SO_PASSPIDFD_THREAD"); + return -1; + } + + /* let the child know the options are set, it can write now */ + if (write(syncpipe[1], "1", 1) != 1) + return -1; + close(syncpipe[1]); + + /* wait until every writer has queued its message */ + if (read(ackpipe[0], &sync, 1) != 1) + return -1; + close(ackpipe[0]); + + for (i = 0; i < nread; i++) + if (read_pidfd_msg(sk[0], &out[i])) + return -1; + + /* release the writers */ + for (i = 0; i < nwriters; i++) + if (write(sk[0], "x", 1) != 1) + return -1; + close(sk[0]); + + waitpid(child, &child_status, 0); + if (!WIFEXITED(child_status) || WEXITSTATUS(child_status)) + return -1; + + return 0; +} + +static int sockopt_set(int fd, int optname, int val) +{ + return setsockopt(fd, SOL_SOCKET, optname, &val, sizeof(val)); +} + +static int sockopt_get(int fd, int optname) +{ + socklen_t len = sizeof(int); + int val = -1; + + if (getsockopt(fd, SOL_SOCKET, optname, &val, &len)) + return -1; + + return val; +} + +TEST(scm_pidfd_setsockopt_values) +{ + int sk[2]; + + ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, sk)); + + /* Verify that the options are truly set independently */ + ASSERT_EQ(0, sockopt_set(sk[0], SO_PASSPIDFD_THREAD, 1)); + ASSERT_EQ(1, sockopt_get(sk[0], SO_PASSPIDFD_THREAD)); + ASSERT_EQ(0, sockopt_get(sk[0], SO_PASSPIDFD)); + + ASSERT_EQ(0, sockopt_set(sk[0], SO_PASSPIDFD, 1)); + ASSERT_EQ(1, sockopt_get(sk[0], SO_PASSPIDFD)); + ASSERT_EQ(1, sockopt_get(sk[0], SO_PASSPIDFD_THREAD)); + + /* Verify that the options are truly cleared independently */ + ASSERT_EQ(0, sockopt_set(sk[0], SO_PASSPIDFD, 0)); + ASSERT_EQ(0, sockopt_get(sk[0], SO_PASSPIDFD)); + ASSERT_EQ(1, sockopt_get(sk[0], SO_PASSPIDFD_THREAD)); + + ASSERT_EQ(0, sockopt_set(sk[0], SO_PASSPIDFD_THREAD, 0)); + ASSERT_EQ(0, sockopt_get(sk[0], SO_PASSPIDFD_THREAD)); + ASSERT_EQ(0, sockopt_get(sk[0], SO_PASSPIDFD)); + + close(sk[0]); + close(sk[1]); +} + +/* A receiver that only asked about the process cannot tell the two + * threads apart, so their writes are glued into one read. + */ +TEST(scm_pidfd_stream_glues_threads) +{ + struct pidfd_msg msg[1] = { 0 }; + + ASSERT_EQ(0, pidfd_flow(send_ids_thread, 2, true, false, msg, 1)); + EXPECT_EQ(sizeof(struct thread_ids) * 2, msg[0].len); + + EXPECT_TRUE(msg[0].have_tgid); + EXPECT_FALSE(msg[0].have_thread); + EXPECT_EQ(msg[0].ids.pid, msg[0].tgid_info.pid); + EXPECT_EQ(msg[0].ids.pid, msg[0].tgid_info.tgid); +} + +/* A receiver asking for both thread and process pidfd should get a unique + * msg for each thread writing + */ +TEST(scm_pidfd_thread_stream_splits_on_threads) +{ + struct pidfd_msg msg[2] = { 0 }; + int i; + + ASSERT_EQ(0, pidfd_flow(send_ids_thread, 2, true, true, msg, 2)); + + for (i = 0; i < 2; i++) { + EXPECT_EQ(sizeof(struct thread_ids), msg[i].len); + EXPECT_TRUE(msg[i].have_tgid); + EXPECT_TRUE(msg[i].have_thread); + + /* Make sure the info makes sense for the pidfd type */ + EXPECT_EQ(msg[i].ids.pid, msg[i].tgid_info.pid); + EXPECT_EQ(msg[i].ids.pid, msg[i].tgid_info.tgid); + EXPECT_EQ(msg[i].ids.tid, msg[i].thread_info.pid); + EXPECT_EQ(msg[i].ids.pid, msg[i].thread_info.tgid); + EXPECT_NE(msg[i].ids.pid, msg[i].ids.tid); + } + + /* Make sure we really got unique threads per message */ + EXPECT_NE(msg[0].ids.tid, msg[1].ids.tid); + EXPECT_EQ(msg[0].ids.pid, msg[1].ids.pid); + EXPECT_EQ(msg[0].tgid_info.pid, msg[1].tgid_info.pid); + EXPECT_NE(msg[0].thread_info.pid, msg[1].thread_info.pid); +} + +/* Sends from the thread-group leader */ +static int leader_flow(bool want_thread, struct pidfd_msg *out) +{ + struct thread_ids ids = { + .pid = getpid(), + .tid = gettid(), + }; + int sk[2]; + int on = 1; + int ret; + + if (socketpair(AF_UNIX, SOCK_STREAM, 0, sk)) + return -1; + + if (setsockopt(sk[0], SOL_SOCKET, SO_PASSPIDFD, &on, sizeof(on))) { + log_err("Failed to set SO_PASSPIDFD"); + return -1; + } + + if (want_thread && + setsockopt(sk[0], SOL_SOCKET, SO_PASSPIDFD_THREAD, &on, sizeof(on))) { + log_err("Failed to set SO_PASSPIDFD_THREAD"); + return -1; + } + + if (send(sk[1], &ids, sizeof(ids), 0) != sizeof(ids)) { + log_err("send"); + return -1; + } + + ret = read_pidfd_msg(sk[0], out); + + close(sk[0]); + close(sk[1]); + + return ret; +} + +TEST(scm_pidfd_leader_sender) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(getpid(), gettid()); + + ASSERT_EQ(0, leader_flow(false, &msg)); + ASSERT_TRUE(msg.have_tgid); + EXPECT_FALSE(msg.have_thread); + + EXPECT_EQ(getpid(), msg.tgid_info.pid); + EXPECT_EQ(getpid(), msg.tgid_info.tgid); + EXPECT_FALSE(msg.tgid_flags & O_EXCL); +} + +TEST(scm_pidfd_thread_leader_sender) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(getpid(), gettid()); + + ASSERT_EQ(0, leader_flow(true, &msg)); + ASSERT_TRUE(msg.have_tgid); + ASSERT_TRUE(msg.have_thread); + + EXPECT_EQ(getpid(), msg.tgid_info.pid); + EXPECT_EQ(getpid(), msg.tgid_info.tgid); + EXPECT_EQ(getpid(), msg.thread_info.pid); + EXPECT_EQ(getpid(), msg.thread_info.tgid); + + EXPECT_FALSE(msg.tgid_flags & O_EXCL); + EXPECT_TRUE(msg.thread_flags & O_EXCL); +} + +TEST(scm_pidfd_thread_and_group) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(0, pidfd_flow(send_ids_thread, 1, true, true, &msg, 1)); + ASSERT_NE(msg.ids.pid, msg.ids.tid); + ASSERT_TRUE(msg.have_tgid); + ASSERT_TRUE(msg.have_thread); + + EXPECT_EQ(msg.ids.pid, msg.tgid_info.pid); + EXPECT_EQ(msg.ids.pid, msg.tgid_info.tgid); + + EXPECT_EQ(msg.ids.tid, msg.thread_info.pid); + EXPECT_EQ(msg.ids.pid, msg.thread_info.tgid); +} + +TEST(scm_pidfd_thread) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(0, pidfd_flow(send_ids_thread, 1, false, true, &msg, 1)); + ASSERT_NE(msg.ids.pid, msg.ids.tid); + ASSERT_TRUE(msg.have_thread); + EXPECT_FALSE(msg.have_tgid); + EXPECT_EQ(msg.ids.tid, msg.thread_info.pid); + EXPECT_EQ(msg.ids.pid, msg.thread_info.tgid); +} + +TEST(scm_pidfd_thread_group) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(0, pidfd_flow(send_ids_thread, 1, true, false, &msg, 1)); + ASSERT_NE(msg.ids.pid, msg.ids.tid); + ASSERT_TRUE(msg.have_tgid); + EXPECT_FALSE(msg.have_thread); + EXPECT_EQ(msg.ids.pid, msg.tgid_info.pid); + EXPECT_EQ(msg.ids.pid, msg.tgid_info.tgid); +} + +TEST(scm_pidfd_thread_creds) +{ + struct pidfd_msg msg = { 0 }; + + ASSERT_EQ(0, pidfd_flow(send_ids_creds_thread, 1, true, true, &msg, 1)); + ASSERT_NE(msg.ids.pid, msg.ids.tid); + ASSERT_TRUE(msg.have_tgid); + ASSERT_TRUE(msg.have_thread); + EXPECT_EQ(msg.ids.pid, msg.tgid_info.pid); + EXPECT_EQ(msg.ids.pid, msg.tgid_info.tgid); + EXPECT_EQ(msg.ids.tid, msg.thread_info.pid); + EXPECT_EQ(msg.ids.pid, msg.thread_info.tgid); +} + +static void *peer_connect_thread(void *arg) +{ + struct sock_addr *sa = arg; + struct thread_ids ids = { + .pid = getpid(), + .tid = gettid(), + }; + int fd; + char sync; + + fd = socket(AF_UNIX, SOCK_STREAM, 0); + if (fd < 0) + return (void *)1; + + if (connect(fd, (struct sockaddr *)&sa->listen_addr, sa->addrlen)) + return (void *)1; + + if (send(fd, &ids, sizeof(ids), 0) != sizeof(ids)) + return (void *)1; + + /* stay alive until the server has looked at our pidfd */ + if (read(fd, &sync, 1) != 1) + return (void *)1; + + close(fd); + return NULL; +} + +static int peer_pidfd_info(int fd, int optname, struct pidfd_info *info) +{ + int pidfd; + socklen_t len = sizeof(pidfd); + + if (getsockopt(fd, SOL_SOCKET, optname, &pidfd, &len)) { + log_err("getsockopt(SO_PEERPIDFD*)"); + return -1; + } + + info->mask = PIDFD_INFO_PID; + if (ioctl(pidfd, PIDFD_GET_INFO, info)) { + log_err("ioctl(PIDFD_GET_INFO)"); + close(pidfd); + return -1; + } + + close(pidfd); + return 0; +} + +/* SO_PEERPIDFD_THREAD returns a pidfd for the peer's connecting thread. */ +TEST(so_peerpidfd_thread) +{ + struct sock_addr sa; + struct thread_ids ids; + struct pidfd_info info; + pthread_t thread; + void *tret; + int server, cfd; + + server = socket(AF_UNIX, SOCK_STREAM, 0); + ASSERT_LE(0, server); + + fill_sockaddr(&sa, true); + ASSERT_EQ(0, bind(server, (struct sockaddr *)&sa.listen_addr, sa.addrlen)); + ASSERT_EQ(0, listen(server, 1)); + + ASSERT_EQ(0, pthread_create(&thread, NULL, peer_connect_thread, &sa)); + + cfd = accept(server, NULL, NULL); + ASSERT_LE(0, cfd); + + ASSERT_EQ(sizeof(ids), recv(cfd, &ids, sizeof(ids), MSG_WAITALL)); + ASSERT_NE(ids.pid, ids.tid); + + /* SO_PEERPIDFD refers to the peer's thread-group. */ + ASSERT_EQ(0, peer_pidfd_info(cfd, SO_PEERPIDFD, &info)); + EXPECT_EQ(ids.pid, info.pid); + EXPECT_EQ(ids.pid, info.tgid); + + /* SO_PEERPIDFD_THREAD refers to the connecting thread. */ + ASSERT_EQ(0, peer_pidfd_info(cfd, SO_PEERPIDFD_THREAD, &info)); + EXPECT_EQ(ids.tid, info.pid); + EXPECT_EQ(ids.pid, info.tgid); + + /* release the connecting thread */ + ASSERT_EQ(1, write(cfd, "x", 1)); + ASSERT_EQ(0, pthread_join(thread, &tret)); + ASSERT_EQ(NULL, tret); + + close(cfd); + close(server); +} + TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile index 6bace04227a743..6981c220eed76b 100644 --- a/tools/testing/selftests/nolibc/Makefile +++ b/tools/testing/selftests/nolibc/Makefile @@ -10,7 +10,7 @@ cc-option = $(call __cc-option, $(CC),,$(1),$(2)) include Makefile.include $(OUTPUT)/nolibc-test: CFLAGS = -nostdlib -nostdinc -static \ - -isystem $(top_srcdir)/tools/include/nolibc -isystem $(top_srcdir)/usr/include \ + -isystem $(top_srcdir)/tools/include/nolibc $(KHDR_INCLUDES) \ $(CFLAGS_NOLIBC_TEST) $(OUTPUT)/nolibc-test: LDLIBS = $(if $(LLVM),,-lgcc) $(OUTPUT)/nolibc-test: $(NOLIBC_TEST_SOURCES) | headers diff --git a/tools/testing/selftests/nolibc/Makefile.include b/tools/testing/selftests/nolibc/Makefile.include index ea520eac64a677..14e51c821d8a8d 100644 --- a/tools/testing/selftests/nolibc/Makefile.include +++ b/tools/testing/selftests/nolibc/Makefile.include @@ -8,6 +8,7 @@ _CFLAGS_SANITIZER ?= $(call cc-option,-fsanitize=undefined -fsanitize-trap=all) CFLAGS_NOLIBC_TEST ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 -ggdb \ -W -Wall -Wextra -Wundef -Wwrite-strings \ $(call cc-option,-fno-stack-protector) $(call cc-option,-Wmissing-prototypes) \ + $(call cc-option,--start-no-unused-arguments) -no-pie $(call cc-option,--end-no-unused-arguments) \ $(_CFLAGS_STACKPROTECTOR) $(_CFLAGS_SANITIZER) NOLIBC_TEST_SOURCES := nolibc-test.c nolibc-test-linkage.c nolibc-test-ignore-errno.c diff --git a/tools/testing/selftests/nolibc/Makefile.nolibc b/tools/testing/selftests/nolibc/Makefile.nolibc index f70c8dfca01869..52924435628016 100644 --- a/tools/testing/selftests/nolibc/Makefile.nolibc +++ b/tools/testing/selftests/nolibc/Makefile.nolibc @@ -65,6 +65,7 @@ ARCH_sparc32 = sparc ARCH_sparc64 = sparc ARCH_sh4 = sh ARCH_parisc32 = parisc +ARCH_parisc64 = parisc64 ARCH := $(or $(ARCH_$(XARCH)),$(XARCH)) # kernel image names by architecture @@ -113,6 +114,7 @@ EXTRACONFIG_sparc32 = -e CONFIG_TMPFS EXTRACONFIG_m68k = -e CONFIG_BLK_DEV_INITRD EXTRACONFIG_sh4 = -e CONFIG_BLK_DEV_INITRD -e CONFIG_CMDLINE_FROM_BOOTLOADER EXTRACONFIG_alpha = -e CONFIG_BLK_DEV_INITRD +EXTRACONFIG_parisc64 = -d CONFIG_COMPAT EXTRACONFIG = $(EXTRACONFIG_$(XARCH)) # optional tests to run (default = all) @@ -134,6 +136,7 @@ QEMU_ARCH_loongarch = loongarch64 QEMU_ARCH_sparc32 = sparc QEMU_ARCH_openrisc = or1k QEMU_ARCH_parisc32 = hppa +QEMU_ARCH_parisc64 = hppa QEMU_ARCH = $(or $(QEMU_ARCH_$(XARCH)),$(XARCH)) QEMU_ARCH_USER_ppc64le = ppc64le @@ -175,6 +178,7 @@ QEMU_ARGS_m68k = -M virt -append "console=ttyGF0,115200 panic=-1 $(TEST:%= QEMU_ARGS_sh4 = -M r2d -serial file:/dev/stdout -append "console=ttySC1,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_openrisc = -M virt -m 512M -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_parisc32 = -M B160L -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" +QEMU_ARGS_parisc64 = -M C3700 -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_alpha = -M clipper -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS = -m 1G $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_BIOS) $(QEMU_ARGS_EXTRA) @@ -193,6 +197,7 @@ CFLAGS_x32 = -mx32 CFLAGS_arm = -marm CFLAGS_armthumb = -mthumb -march=armv6t2 CFLAGS_parisc32 = -mfast-indirect-calls +CFLAGS_parisc64 = -Wl,--no-dynamic-linker -Wl,--entry=_start -O2 CFLAGS_ppc = -m32 -mbig-endian -mno-vsx $(call cc-option,-mmultiple) CFLAGS_ppc64 = -m64 -mbig-endian -mno-vsx $(call cc-option,-mmultiple) CFLAGS_ppc64le = -m64 -mlittle-endian -mno-vsx $(call cc-option,-mabi=elfv2) @@ -206,6 +211,8 @@ CFLAGS_mips64be = -EB -mabi=64 -march=mips64r2 CFLAGS_loongarch = $(if $(LLVM),-fuse-ld=lld) CFLAGS_sparc32 = $(call cc-option,-m32) -mcpu=v8 CFLAGS_sh4 = -ml -m4 +# github.com/llvm/llvm-project/issues/216765 +CFLAGS_hexagon = -fno-sanitize=undefined ifeq ($(origin XARCH),command line) CFLAGS_XARCH = $(CFLAGS_$(XARCH)) endif @@ -222,10 +229,16 @@ LDFLAGS := # Modify CFLAGS based on LLVM= include $(srctree)/tools/scripts/Makefile.include +ifeq ($(ARCH),hexagon) +# tools/scripts/Makefile.include requires CROSS_COMPILE which does not exist for hexagon +CFLAGS += --target=hexagon-linux-musl +CLANG_CROSS_FLAGS += --target=hexagon-linux-musl +endif + REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{if (!f) printf("\n"); f++; print;} /\[SKIPPED\][\r]*$$/{s++} \ /^Total number of errors:/{done++} \ END{ printf("\n%3d test(s): %3d passed, %3d skipped, %3d failed => status: ", p+s+f, p, s, f); \ - if (f || !p || !done) printf("failure\n"); else if (s) printf("warning\n"); else printf("success\n");; \ + if (f || !p || !done) printf("failure\n"); else printf("success\n");; \ printf("\nSee all results in %s\n", ARGV[1]); }' # Execute the toplevel kernel Makefile @@ -295,7 +308,7 @@ run-user: nolibc-test $(Q)$(REPORT) $(CURDIR)/run.out initramfs.cpio: kernel nolibc-test - $(QUIET_GEN)echo 'file /init nolibc-test 755 0 0' | $(objtree)/usr/gen_init_cpio - > initramfs.cpio + $(QUIET_GEN)echo 'file /init nolibc-test 755 0 0' | $(objtree)/scripts/gen_init_cpio - > initramfs.cpio initramfs: nolibc-test $(QUIET_MKDIR)mkdir -p initramfs diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c index ed860b0a15a1e5..f3d0d27cdbe178 100644 --- a/tools/testing/selftests/nolibc/nolibc-test.c +++ b/tools/testing/selftests/nolibc/nolibc-test.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -1368,6 +1369,51 @@ int test_pipe(void) return !!memcmp(buf, msg, len); } +int test_pwrite_pread(void) +{ + const off_t offset = (0x12ULL << 32) | 0x34; + ssize_t ret; + char buf[2]; + int fd, err; + off_t offs; + + fd = open("/tmp", O_TMPFILE | O_RDWR, 0644); + if (fd == -1) + return __LINE__; + + /* Write into the middle of the file. */ + ret = pwrite(fd, "abc", 3, offset); + if (ret != 3) { + err = __LINE__; + goto out; + } + + /* Make sure the file offset points somewhere else before pread(). */ + offs = lseek(fd, 0, SEEK_END); + if (offs != offset + 3) { + err = __LINE__; + goto out; + } + + ret = pread(fd, buf, 2, offset + 1); + if (ret != 2) { + err = __LINE__; + goto out; + } + + if (buf[0] != 'b' || buf[1] != 'c') { + err = __LINE__; + goto out; + } + + err = 0; + +out: + close(fd); + + return err; +} + int test_rlimit(void) { struct rlimit rlim = { @@ -1524,12 +1570,6 @@ int test_large_file(void) ssize_t written; off_t off; -#if defined(__mips__) && defined(_ABIN32) - /* https://lore.kernel.org/qemu-devel/fed03914-a95a-4522-a432-f129264cb2ac@t-8ch.de/ */ - if (getpid() != 1) - return 0; -#endif - if (large_seek < UINT32_MAX) { errno = EOVERFLOW; return -1; @@ -1563,6 +1603,72 @@ int test_large_file(void) return ret; } +int test_sendfile(void) +{ + char data_in[] = "This is some data"; + const size_t data_sz = sizeof(data_in); + char data_out[data_sz + 10]; + int in_fd, out_fd; + int ret; + + /* Create two tmp files */ + in_fd = open("/tmp", O_TMPFILE | O_RDWR, 0644); + out_fd = open("/tmp", O_TMPFILE | O_RDWR, 0644); + + if (in_fd == -1 || out_fd == -1) { + ret = __LINE__; + goto out; + } + + /* Populate the "in" file */ + ret = write(in_fd, data_in, data_sz); + if (ret != data_sz) { + ret = __LINE__; + goto out; + } + + /* Rewind the "in" file */ + if (lseek(in_fd, 0, SEEK_SET)) { + ret = __LINE__; + goto out; + } + + /* Use sendfile() to copy "in" to "out" */ + ret = sendfile(out_fd, in_fd, NULL, data_sz + 5); + if (ret != data_sz) { + ret = __LINE__; + goto out; + } + + /* Rewind the "out" file */ + if (lseek(out_fd, 0, SEEK_SET)) { + ret = __LINE__; + goto out; + } + + /* Read back the transferred data */ + ret = read(out_fd, data_out, sizeof(data_out)); + if (ret != data_sz) { + ret = __LINE__; + goto out; + } + + /* Check we have the same data in both files */ + if (memcmp(data_out, data_in, data_sz)) { + ret = __LINE__; + goto out; + } + + /* Test passed */ + ret = 0; + +out: + close(out_fd); + close(in_fd); + + return ret; +} + /* Run syscall tests between IDs and . * Return 0 on success, non-zero on failure. */ @@ -1584,6 +1690,7 @@ int run_syscall(int min, int max) void *p1, *p2; int has_gettid = 1; int has_brk; + int has_working_64bit_syscall_arg = 1; /* indicates whether or not /proc is mounted */ proc = stat("/proc", &stat_buf) == 0; @@ -1599,6 +1706,11 @@ int run_syscall(int min, int max) /* on musl setting brk()/sbrk() always fails */ has_brk = brk(0) == 0; +#if defined(__mips__) && defined(_ABIN32) + /* https://lore.kernel.org/qemu-devel/fed03914-a95a-4522-a432-f129264cb2ac@t-8ch.de/ */ + has_working_64bit_syscall_arg = getpid() == 1; +#endif + for (test = min; test >= 0 && test <= max; test++) { int llen = 0; /* line length */ @@ -1644,6 +1756,7 @@ int run_syscall(int min, int max) CASE_TEST(execve_root); EXPECT_SYSER(1, execve("/", (char*[]){ [0] = (char []){"/"}, [1] = NULL }, NULL), -1, EACCES); break; CASE_TEST(fchdir_stdin); EXPECT_SYSER(1, fchdir(STDIN_FILENO), -1, ENOTDIR); break; CASE_TEST(fchdir_badfd); EXPECT_SYSER(1, fchdir(-1), -1, EBADF); break; + CASE_TEST(fdopendir_notdir); EXPECT_SYSER(1, (uintptr_t)fdopendir(STDIN_FILENO), (uintptr_t)NULL, ENOTDIR); break; CASE_TEST(file_stream); EXPECT_SYSZR(1, test_file_stream()); break; CASE_TEST(file_stream_wsr); EXPECT_SYSZR(1, test_file_stream_wsr()); break; CASE_TEST(fork); EXPECT_SYSZR(1, test_fork(FORK_STANDARD)); break; @@ -1672,11 +1785,13 @@ int run_syscall(int min, int max) CASE_TEST(open_blah); EXPECT_SYSER(1, tmp = open("/proc/self/blah", O_RDONLY), -1, ENOENT); if (tmp != -1) close(tmp); break; CASE_TEST(openat_dir); EXPECT_SYSZR(1, test_openat()); break; CASE_TEST(open_mode); EXPECT_SYSZR(1, test_open_mode()); break; + CASE_TEST(opendir_notdir); EXPECT_SYSER(1, (uintptr_t)opendir("/dev/null"), (uintptr_t)NULL, ENOTDIR); break; CASE_TEST(pipe); EXPECT_SYSZR(1, test_pipe()); break; CASE_TEST(poll_null); EXPECT_SYSZR(1, poll(NULL, 0, 0)); break; CASE_TEST(poll_stdout); EXPECT_SYSNE(1, ({ struct pollfd fds = { 1, POLLOUT, 0}; poll(&fds, 1, 0); }), -1); break; CASE_TEST(poll_fault); EXPECT_SYSER(1, poll(NULL, 1, 0), -1, EFAULT); break; CASE_TEST(prctl); EXPECT_SYSER(1, prctl(PR_SET_NAME, (unsigned long)NULL, 0, 0, 0), -1, EFAULT); break; + CASE_TEST(pwrite_pread); EXPECT_SYSZR(has_working_64bit_syscall_arg, test_pwrite_pread()); break; CASE_TEST(read_badf); EXPECT_SYSER(1, read(-1, &tmp, 1), -1, EBADF); break; CASE_TEST(rlimit); EXPECT_SYSZR(1, test_rlimit()); break; CASE_TEST(rmdir_blah); EXPECT_SYSER(1, rmdir("/blah"), -1, ENOENT); break; @@ -1684,6 +1799,7 @@ int run_syscall(int min, int max) CASE_TEST(select_null); EXPECT_SYSZR(1, ({ struct timeval tv = { 0 }; select(0, NULL, NULL, NULL, &tv); })); break; CASE_TEST(select_stdout); EXPECT_SYSNE(1, ({ fd_set fds; FD_ZERO(&fds); FD_SET(1, &fds); select(2, NULL, &fds, NULL, NULL); }), -1); break; CASE_TEST(select_fault); EXPECT_SYSER(1, select(1, (void *)1, NULL, NULL, 0), -1, EFAULT); break; + CASE_TEST(sendfile); EXPECT_SYSZR(1, test_sendfile()); break; CASE_TEST(stat_blah); EXPECT_SYSER(1, stat("/proc/self/blah", &stat_buf), -1, ENOENT); break; CASE_TEST(stat_fault); EXPECT_SYSER(1, stat(NULL, &stat_buf), -1, EFAULT); break; CASE_TEST(stat_rdev); EXPECT_SYSZR(1, ({ int ret = stat("/dev/null", &stat_buf); ret ?: stat_buf.st_rdev != makedev(1, 3); })); break; @@ -1711,7 +1827,7 @@ int run_syscall(int min, int max) CASE_TEST(_syscall_noargs); EXPECT_SYSEQ(is_nolibc, _syscall(__NR_getpid), getpid()); break; CASE_TEST(_syscall_args); EXPECT_SYSEQ(is_nolibc, _syscall(__NR_statx, 0, NULL, 0, 0, NULL), -EFAULT); break; CASE_TEST(namespace); EXPECT_SYSZR(euid0 && proc, test_namespace()); break; - CASE_TEST(largefile); EXPECT_SYSZR(1, test_large_file()); break; + CASE_TEST(largefile); EXPECT_SYSZR(has_working_64bit_syscall_arg, test_large_file()); break; case __LINE__: return ret; /* must be last */ /* note: do not set any defaults so as to permit holes above */ @@ -1771,6 +1887,26 @@ int test_time_types(void) return 0; } +int test_fd_set(void) +{ + fd_set set, clr; + int fd, i; + + /* FD_SET() and FD_CLR() must change exactly one fd */ + for (fd = 0; fd < FD_SETSIZE; fd++) { + FD_ZERO(&set); + FD_SET(fd, &set); + memset(&clr, 0xff, sizeof(clr)); + FD_CLR(fd, &clr); + for (i = 0; i < FD_SETSIZE; i++) + if (!!FD_ISSET(i, &set) != (i == fd) || + !!FD_ISSET(i, &clr) != (i != fd)) + return 1; + } + + return 0; +} + int test_malloc(void) { size_t sz_array1, sz_array2, sz_array3; @@ -1952,6 +2088,7 @@ int run_stdlib(int min, int max) CASE_TEST(memchr_foobar6_o); EXPECT_STREQ(1, memchr("foobar", 'o', 6), "oobar"); break; CASE_TEST(memchr_foobar3_b); EXPECT_STRZR(1, memchr("foobar", 'b', 3)); break; CASE_TEST(time_types); EXPECT_ZR(is_nolibc, test_time_types()); break; + CASE_TEST(fd_set); EXPECT_ZR(1, test_fd_set()); break; CASE_TEST(makedev); EXPECT_EQ(1, makedev(0x12, 0x34), 0x1234); break; CASE_TEST(major); EXPECT_EQ(1, major(0x1234), 0x12); break; CASE_TEST(minor); EXPECT_EQ(1, minor(0x1234), 0x34); break; @@ -1968,6 +2105,14 @@ int run_stdlib(int min, int max) CASE_TEST(htole32); EXPECT_EQ(1, htole32(is_le ? 0x01234567 : 0x67452301), 0x01234567); break; CASE_TEST(htobe64); EXPECT_EQ(1, htobe64(is_le ? 0x0123456789000000 : 0x8967452301), 0x8967452301); break; CASE_TEST(htole64); EXPECT_EQ(1, htole64(is_le ? 0x0123456789 : 0x8967452301000000), 0x0123456789); break; + CASE_TEST(wexitstatus); EXPECT_EQ(1, WEXITSTATUS(0x1200), 0x12); break; + CASE_TEST(wifexited_exit0); EXPECT_EQ(1, WIFEXITED(0), 1); break; + CASE_TEST(wifexited_signal); EXPECT_EQ(1, WIFEXITED(SIGKILL), 0); break; + CASE_TEST(wifsignaled_exit0); EXPECT_EQ(1, WIFSIGNALED(0), 0); break; + CASE_TEST(wifsignaled_exit); EXPECT_EQ(1, WIFSIGNALED(0x1200), 0); break; + CASE_TEST(wifsignaled_signal); EXPECT_EQ(1, WIFSIGNALED(SIGKILL), 1); break; + CASE_TEST(wifsignaled_stopped); EXPECT_EQ(1, WIFSIGNALED((SIGSTOP << 8) | 0x7f), 0); break; + CASE_TEST(wtermsig_core); EXPECT_EQ(1, WTERMSIG(0x80 | SIGSEGV), SIGSEGV); break; case __LINE__: return ret; /* must be last */ diff --git a/tools/testing/selftests/nolibc/run-tests.sh b/tools/testing/selftests/nolibc/run-tests.sh index dc0b1649c6419e..2844fed26df177 100755 --- a/tools/testing/selftests/nolibc/run-tests.sh +++ b/tools/testing/selftests/nolibc/run-tests.sh @@ -29,8 +29,9 @@ all_archs=( sparc32 sparc64 m68k sh4 - parisc32 + parisc32 parisc64 alpha + hexagon ) archs="${all_archs[@]}" @@ -121,6 +122,7 @@ crosstool_arch() { sparc*) echo sparc64;; x32*) echo x86_64;; parisc32) echo hppa;; + parisc64) echo hppa64;; *) echo "$1";; esac } @@ -132,6 +134,13 @@ crosstool_abi() { esac } +need_gcc() { + case "$1" in + hexagon);; + *) echo "1";; + esac +} + download_crosstool() { arch="$(crosstool_arch "$1")" abi="$(crosstool_abi "$1")" @@ -164,14 +173,18 @@ test_arch() { arch=$1 ct_arch=$(crosstool_arch "$arch") ct_abi=$(crosstool_abi "$1") + gcc="$(need_gcc "$1")" + cross_compile= - if [ ! -d "${download_location}gcc-${crosstool_version}-nolibc/${ct_arch}-${ct_abi}/bin/." ]; then + if [ -n "$gcc" ] && [ ! -d "${download_location}gcc-${crosstool_version}-nolibc/${ct_arch}-${ct_abi}/bin/." ]; then echo "No toolchain found in ${download_location}gcc-${crosstool_version}-nolibc/${ct_arch}-${ct_abi}." echo "Did you install the toolchains or set the correct arch ? Rerun with -h for help." return 1 fi - cross_compile=$(realpath "${download_location}gcc-${crosstool_version}-nolibc/${ct_arch}-${ct_abi}/bin/${ct_arch}-${ct_abi}-") + if [ -n "$gcc" ]; then + cross_compile=$(realpath "${download_location}gcc-${crosstool_version}-nolibc/${ct_arch}-${ct_abi}/bin/${ct_arch}-${ct_abi}-") + fi build_dir="${build_location}/${arch}" if [ "$werror" -ne 0 ]; then CFLAGS_EXTRA="$CFLAGS_EXTRA -Werror -Wl,--fatal-warnings" @@ -194,11 +207,18 @@ test_arch() { exit 1 esac printf '%-15s' "$arch:" - if [ "$arch" = "m68k" -o "$arch" = "sh4" -o "$arch" = "openrisc" -o "$arch" = "parisc32" -o "$arch" = "alpha" ] && [ "$llvm" = "1" ]; then + if [ "$llvm" = "1" ] && \ + [ "$arch" = "m68k" -o "$arch" = "sh4" -o "$arch" = "openrisc" \ + -o "$arch" == "parisc32" -o "$arch" == "parisc64" -o "$arch" = "alpha" \ + ]; then + echo "Unsupported configuration" + return + fi + if [ "$test_mode" = "user" ] && [ "$arch" = "x32" -o "$arch" = "parisc64" ]; then echo "Unsupported configuration" return fi - if [ "$arch" = "x32" ] && [ "$test_mode" = "user" ]; then + if [ -z "$llvm" -o "$test_mode" = "system" ] && [ "$arch" = "hexagon" ]; then echo "Unsupported configuration" return fi diff --git a/tools/testing/selftests/pid_namespace/.gitignore b/tools/testing/selftests/pid_namespace/.gitignore index c647c6eb336728..b87295802a73cb 100644 --- a/tools/testing/selftests/pid_namespace/.gitignore +++ b/tools/testing/selftests/pid_namespace/.gitignore @@ -1,3 +1,4 @@ pid_max +pidns_dead_ancestor pidns_init_via_setns regression_enomem diff --git a/tools/testing/selftests/pid_namespace/Makefile b/tools/testing/selftests/pid_namespace/Makefile index b01a924ac04bd8..3199e6bebbdd80 100644 --- a/tools/testing/selftests/pid_namespace/Makefile +++ b/tools/testing/selftests/pid_namespace/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 CFLAGS += -g $(KHDR_INCLUDES) -TEST_GEN_PROGS = regression_enomem pid_max pidns_init_via_setns +TEST_GEN_PROGS = regression_enomem pid_max pidns_dead_ancestor pidns_init_via_setns LOCAL_HDRS += $(selfdir)/pidfd/pidfd.h diff --git a/tools/testing/selftests/pid_namespace/pidns_dead_ancestor.c b/tools/testing/selftests/pid_namespace/pidns_dead_ancestor.c new file mode 100644 index 00000000000000..7524d4975ec752 --- /dev/null +++ b/tools/testing/selftests/pid_namespace/pidns_dead_ancestor.c @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kselftest_harness.h" + +#define STACK_SIZE (1024 * 1024) + +struct pidns_init_args { + int ready_read; + int ready_write; + int release_read; + int release_write; +}; + +static int pidns_init(void *data) +{ + struct pidns_init_args *args = data; + char release; + int error = 0; + + close(args->ready_read); + close(args->release_write); + if (unshare(CLONE_NEWPID)) + error = errno; + if (write(args->ready_write, &error, sizeof(error)) != sizeof(error)) + return 1; + close(args->ready_write); + if (error) + return 0; + if (read(args->release_read, &release, sizeof(release)) != sizeof(release)) + return 1; + close(args->release_read); + return 0; +} + +static int create_unborn_pidns(void) +{ + struct pidns_init_args args; + char path[64], release = 0; + void *stack; + pid_t child; + int ready[2], release_pipe[2], error, status, nsfd = -1; + ssize_t n; + + if (pipe(ready)) + return -1; + if (pipe(release_pipe)) { + error = errno; + close(ready[0]); + close(ready[1]); + errno = error; + return -1; + } + + stack = mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0); + if (stack == MAP_FAILED) { + error = errno; + close(ready[0]); + close(ready[1]); + close(release_pipe[0]); + close(release_pipe[1]); + errno = error; + return -1; + } + + args = (struct pidns_init_args) { + .ready_read = ready[0], + .ready_write = ready[1], + .release_read = release_pipe[0], + .release_write = release_pipe[1], + }; + child = clone(pidns_init, stack + STACK_SIZE, CLONE_NEWPID | SIGCHLD, + &args); + if (child < 0) + goto err_munmap; + + close(ready[1]); + close(release_pipe[0]); + n = read(ready[0], &error, sizeof(error)); + if (n != sizeof(error)) { + error = n < 0 ? errno : EIO; + goto release_child; + } + if (error) + goto release_child; + + snprintf(path, sizeof(path), "/proc/%d/ns/pid_for_children", child); + nsfd = open(path, O_RDONLY | O_CLOEXEC); + if (nsfd < 0) { + error = errno; + goto release_child; + } + +release_child: + write(release_pipe[1], &release, sizeof(release)); + close(release_pipe[1]); + close(ready[0]); + if (waitpid(child, &status, 0) != child || !WIFEXITED(status) || + WEXITSTATUS(status)) { + if (!error) + error = ECHILD; + if (nsfd >= 0) + close(nsfd); + nsfd = -1; + } + munmap(stack, STACK_SIZE); + if (error) { + errno = error; + return -1; + } + return nsfd; + +err_munmap: + error = errno; + munmap(stack, STACK_SIZE); + close(ready[0]); + close(ready[1]); + close(release_pipe[0]); + close(release_pipe[1]); + errno = error; + return -1; +} + +TEST(pidns_dead_ancestor_rejects_fork) +{ + int original_fd, target_fd, fork_errno, ret; + pid_t child; + + target_fd = create_unborn_pidns(); + if (target_fd < 0) { + if (errno == ENOENT) + SKIP(return, "PID namespace handle unavailable through procfs\n"); + if (errno == EPERM || errno == EACCES || errno == EINVAL) + SKIP(return, "PID namespace creation requires privileges\n"); + ASSERT_GE(target_fd, 0); + } + + original_fd = open("/proc/self/ns/pid_for_children", O_RDONLY | O_CLOEXEC); + ASSERT_GE(original_fd, 0); + ret = setns(target_fd, CLONE_NEWPID); + if (ret) { + if (errno == EPERM || errno == EACCES) + SKIP(return, "setns into PID namespace requires privileges\n"); + } + ASSERT_EQ(0, ret); + + child = fork(); + fork_errno = errno; + if (child == 0) + _exit(0); + + if (child > 0) + ASSERT_EQ(child, waitpid(child, NULL, 0)); + ret = setns(original_fd, CLONE_NEWPID); + close(original_fd); + close(target_fd); + ASSERT_EQ(0, ret); + + ASSERT_EQ(-1, child); + ASSERT_EQ(ENOMEM, fork_errno); +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/pidfd/pidfd_open_test.c b/tools/testing/selftests/pidfd/pidfd_open_test.c index 318e6f09c8e0b6..c6698b7cdb479d 100644 --- a/tools/testing/selftests/pidfd/pidfd_open_test.c +++ b/tools/testing/selftests/pidfd/pidfd_open_test.c @@ -168,7 +168,7 @@ int main(int argc, char **argv) } if (info.ppid != getppid()) { ksft_print_msg("ppid %d does not match ppid from ioctl %d\n", - pid, info.pid); + getppid(), info.ppid); goto on_error; } if (info.ruid != getuid()) { diff --git a/tools/testing/selftests/rcutorture/bin/kvm-again.sh b/tools/testing/selftests/rcutorture/bin/kvm-again.sh index b5239b52cb5da7..9060b3dc892626 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm-again.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm-again.sh @@ -180,6 +180,9 @@ fi echo ---- Re-run results directory: $rundir +qemu_binary="$(find "$oldrun" -maxdepth 2 -name 'qemu-cmd' -type f 2>/dev/null | head -1 | xargs -r grep -v '^#' 2>/dev/null | awk 'NF { print $1; exit }')" +boot_image="`identify_boot_image "$qemu_binary"`" + if test "$oldrun" != "$rundir" then # Copy old run directory tree over and adjust. @@ -189,7 +192,7 @@ then echo "Cannot copy from $oldrun to $rundir." usage fi - rm -f "$rundir"/*/{console.log,console.log.diags,qemu_pid,qemu-pid,qemu-retval,Warnings,kvm-test-1-run.sh.out,kvm-test-1-run-qemu.sh.out,vmlinux} "$rundir"/log + rm -f "$rundir"/*/{console.log,console.log.diags,qemu_pid,qemu-pid,qemu-retval,Warnings,kvm-test-1-run.sh.out,kvm-test-1-run-qemu.sh.out} "$rundir"/log touch "$rundir/log" echo $scriptname $args | tee -a "$rundir/log" echo $oldrun > "$rundir/re-run" @@ -197,6 +200,10 @@ then then $arg_link "$oldrun/../../bin" "$rundir/../.." fi + if test "`basename "$boot_image"`" != vmlinux + then + rm -f "$rundir"/*/vmlinux + fi else # Check for a run having already happened. find "$rundir" -name console.log -print > $T/oldrun-console.log @@ -217,7 +224,7 @@ do qemu_cmd_dir="`dirname "$i"`" kernel_dir="`echo $qemu_cmd_dir | sed -e 's/\.[0-9]\+$//'`" jitter_dir="`dirname "$kernel_dir"`" - kvm-transform.sh "$kernel_dir/bzImage" "$qemu_cmd_dir/console.log" "$jitter_dir" "$dur" "$bootargs" < $T/qemu-cmd > $i + kvm-transform.sh "$kernel_dir/`basename $boot_image`" "$qemu_cmd_dir/console.log" "$jitter_dir" "$dur" "$bootargs" < $T/qemu-cmd > $i if test -n "$arg_remote" then echo "# TORTURE_KCONFIG_GDB_ARG=''" >> $i diff --git a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh index 48a8052d5dae37..a8397f86e49dc7 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh @@ -6,6 +6,11 @@ # Usage: kvm-remote.sh "systems" [ ] # kvm-remote.sh "systems" /path/to/old/run [ ] # +# The caller may set the KVM_REMOTE_SSH environment in order to specify +# an alternative ssh command, which is necessary in some environments +# for authentication purposes. This alternative ssn command must support +# ssh's usual arguments. +# # Copyright (C) 2021 Facebook, Inc. # # Authors: Paul E. McKenney @@ -13,6 +18,11 @@ scriptname=$0 args="$*" +if test -z "${KVM_REMOTE_SSH}" +then + KVM_REMOTE_SSH=ssh; export KVM_REMOTE_SSH +fi + if ! test -d tools/testing/selftests/rcutorture/bin then echo $scriptname must be run from top-level directory of kernel source tree. @@ -137,7 +147,7 @@ chmod +x $T/bin/kvm-remote-*.sh # Check first to avoid the need for cleanup for system-name typos for i in $systems do - ssh -o BatchMode=yes $i getconf _NPROCESSORS_ONLN > $T/ssh.stdout 2> $T/ssh.stderr + ${KVM_REMOTE_SSH} -o BatchMode=yes $i getconf _NPROCESSORS_ONLN > $T/ssh.stdout 2> $T/ssh.stderr ret=$? if test "$ret" -ne 0 then @@ -158,14 +168,14 @@ echo Build-products tarball: `du -h $T/binres.tgz` | tee -a "$oldrun/remote-log" for i in $systems do echo Downloading tarball to $i `date` | tee -a "$oldrun/remote-log" - cat $T/binres.tgz | ssh -o BatchMode=yes $i "cd /tmp; tar -xzf -" + cat $T/binres.tgz | ${KVM_REMOTE_SSH} -o BatchMode=yes $i "cd /tmp; tar -xzf -" ret=$? tries=0 while test "$ret" -ne 0 do echo Unable to download $T/binres.tgz to system $i, waiting and then retrying. $tries prior retries. | tee -a "$oldrun/remote-log" sleep 60 - cat $T/binres.tgz | ssh -o BatchMode=yes $i "cd /tmp; tar -xzf -" + cat $T/binres.tgz | ${KVM_REMOTE_SSH} -o BatchMode=yes $i "cd /tmp; tar -xzf -" ret=$? if test "$ret" -ne 0 then @@ -191,7 +201,7 @@ checkremotefile () { while : do - ssh -o BatchMode=yes $1 "test -f \"$2\"" + ${KVM_REMOTE_SSH} -o BatchMode=yes $1 "test -f \"$2\"" ret=$? if test "$ret" -eq 255 then @@ -239,7 +249,7 @@ startbatches () { then continue # System still running last test, skip. fi - ssh -o BatchMode=yes "$i" "cd \"$resdir/$ds\"; touch remote.run; PATH=\"$T/bin:$PATH\" nohup kvm-remote-$curbatch.sh > kvm-remote-$curbatch.sh.out 2>&1 &" 1>&2 + ${KVM_REMOTE_SSH} -o BatchMode=yes "$i" "cd \"$resdir/$ds\"; touch remote.run; PATH=\"$T/bin:$PATH\" nohup kvm-remote-$curbatch.sh > kvm-remote-$curbatch.sh.out 2>&1 &" 1>&2 ret=$? if test "$ret" -ne 0 then @@ -281,7 +291,7 @@ do if test "$ret" -eq 1 then echo " ---" Collecting results from $i `date` | tee -a "$oldrun/remote-log" - ( cd "$oldrun"; ssh -o BatchMode=yes $i "cd $rundir; tar -czf - kvm-remote-*.sh.out */console.log */kvm-test-1-run*.sh.out */qemu[_-]pid */qemu-retval */qemu-affinity; rm -rf $T > /dev/null 2>&1" | tar -xzf - ) + ( cd "$oldrun"; ${KVM_REMOTE_SSH} -o BatchMode=yes $i "cd $rundir; tar -czf - kvm-remote-*.sh.out */console.log */kvm-test-1-run*.sh.out */qemu[_-]pid */qemu-retval */qemu-affinity; rm -rf $T > /dev/null 2>&1" | tar -xzf - ) break; fi if test "$ret" -eq 255 diff --git a/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh b/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh index 4e98c697def487..c29a39f7d8914d 100755 --- a/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh +++ b/tools/testing/selftests/rcutorture/bin/srcu_lockdep.sh @@ -44,7 +44,7 @@ nerrs=0 # Test lockdep's handling of deadlocks. for d in 0 1 do - for t in 0 1 2 + for t in 0 1 2 4 5 do for c in 1 2 3 do @@ -56,7 +56,6 @@ do if ! grep -q '^CONFIG_PROVE_LOCKING=y' .config then echo "rcu_torture_init_srcu_lockdep:Error: CONFIG_PROVE_LOCKING disabled in rcutorture SRCU-P scenario" - nerrs=$((nerrs+1)) err=1 fi if test "$d" -ne 0 && test "$ret" -eq 0 @@ -79,6 +78,79 @@ do done done +# Verify that synchronize_srcu_atomic() does not trigger lockdep +# warnings when called inside rcu_read_lock(). +for c in 1 2 3 +do + err= + val=$((6*10+c)) + tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s \ + --configs "SRCU-P" \ + --kconfig "CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y" \ + --bootargs "rcutorture.test_srcu_lockdep=$val" \ + --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 + ret=$? + mv "$T/kvm.sh.out" "$RCUTORTURE/res/$ds/$val" + if ! grep -q '^CONFIG_PROVE_LOCKING=y' .config + then + echo "rcu_torture_init_srcu_lockdep:Error: CONFIG_PROVE_LOCKING" \ + "disabled in rcutorture SRCU-P scenario" + err=1 + fi + if test "$ret" -ne 0 + then + err=1 + echo -n Unexpected failure for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + elif grep -qE "WARNING: possible (recursive locking|circular locking dependency)" \ + "$RCUTORTURE/res/$ds/$val/SRCU-P/console.log" + then + err=1 + echo -n Unexpected lockdep warning for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + fi + if test -n "$err" + then + grep "rcu_torture_init_srcu_lockdep: test_srcu_lockdep = " "$RCUTORTURE/res/$ds/$val/SRCU-P/console.log" | sed -e 's/^.*rcu_torture_init_srcu_lockdep://' >> "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + cat "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + nerrs=$((nerrs+1)) + fi +done + +# Verify that synchronize_srcu_atomic() detects IRQ context mismatch +# when SRCU reader previously ran with IRQs enabled. +for c in 1 2 3 +do + err= + val=$((1000+7*10+c)) + tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s \ + --configs "SRCU-P" \ + --kconfig "CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y" \ + --bootargs "rcutorture.test_srcu_lockdep=$val" \ + --trust-make --datestamp "$ds/$val" > "$T/kvm.sh.out" 2>&1 + ret=$? + mv "$T/kvm.sh.out" "$RCUTORTURE/res/$ds/$val" + if ! grep -q '^CONFIG_PROVE_LOCKING=y' .config + then + echo "rcu_torture_init_srcu_lockdep:Error: CONFIG_PROVE_LOCKING" \ + "disabled in rcutorture SRCU-P scenario" + err=1 + fi + if test "$ret" -eq 0 + then + err=1 + echo -n Missing lockdep warning for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + elif ! grep -q "inconsistent {HARDIRQ-ON-R}" "$RCUTORTURE/res/$ds/$val/SRCU-P/console.log" + then + err=1 + echo -n Missing lockdep warning for > "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + fi + if test -n "$err" + then + grep "rcu_torture_init_srcu_lockdep: test_srcu_lockdep = " "$RCUTORTURE/res/$ds/$val/SRCU-P/console.log" | sed -e 's/^.*rcu_torture_init_srcu_lockdep://' >> "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + cat "$RCUTORTURE/res/$ds/$val/kvm.sh.err" + nerrs=$((nerrs+1)) + fi +done + # Test lockdep-enabled testing of mixed SRCU readers. for val in 0x1 0xf do @@ -89,7 +161,6 @@ do if ! grep -q '^CONFIG_PROVE_LOCKING=y' .config then echo "rcu_torture_init_srcu_lockdep:Error: CONFIG_PROVE_LOCKING disabled in rcutorture SRCU-P scenario" - nerrs=$((nerrs+1)) err=1 fi if test "$val" = 0xf && test "$ret" -eq 0 diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh index f0083891ee8147..e7b578ec152f16 100755 --- a/tools/testing/selftests/rcutorture/bin/torture.sh +++ b/tools/testing/selftests/rcutorture/bin/torture.sh @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}" do_rt=yes do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided do_srcu_lockdep=yes +do_atomic_srcu=yes do_rcu_rust=no # doyesno - Helper function for yes/no arguments @@ -103,6 +104,7 @@ usage () { echo " --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust" echo " --do-scftorture / --do-no-scftorture / --no-scftorture" echo " --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep" + echo " --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu" echo " --duration [ | h | d ]" echo " --guest-cpu-limit N" echo " --kcsan-kmake-arg kernel-make-arguments" @@ -148,6 +150,7 @@ do do_kcsan=yes do_clocksourcewd="${ifnotaarch64}" do_srcu_lockdep=yes + do_atomic_srcu=yes ;; --do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig) do_allmodconfig=`doyesno "$1" --do-allmodconfig` @@ -183,6 +186,7 @@ do do_kcsan=no do_clocksourcewd=no do_srcu_lockdep=no + do_atomic_srcu=no ;; --do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm) do_normal=`doyesno "$1" --do-normal` @@ -212,6 +216,9 @@ do --do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep) do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep` ;; + --do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu) + do_atomic_srcu=`doyesno "$1" --do-atomic-srcu` + ;; --duration) checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error' mult=1 @@ -497,6 +504,23 @@ then torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make fi +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU +# (SRCU-T). The reader flavor selects srcu_read_lock_atomic() and +# synchronize_srcu_atomic(). Tiny SRCU requires SMP=n, which aarch64 +# does not support. +if test "$do_atomic_srcu" = "yes" +then + torture_bootargs="rcutorture.reader_flavor=0x10" + configs_atomic_srcu="SRCU-N SRCU-P" + if test "$ifnotaarch64" = yes + then + configs_atomic_srcu="$configs_atomic_srcu SRCU-T" + fi + torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \ + --allcpus --duration "$duration_rcutorture" \ + --configs "$configs_atomic_srcu" --trust-make +fi + if test "$do_locktorture" = "yes" then torture_bootargs="torture.disable_onoff_at_boot" diff --git a/tools/testing/selftests/rseq/rseq-arm64.h b/tools/testing/selftests/rseq/rseq-arm64.h index 21e1626a723584..f9fecf941bf22d 100644 --- a/tools/testing/selftests/rseq/rseq-arm64.h +++ b/tools/testing/selftests/rseq/rseq-arm64.h @@ -12,12 +12,7 @@ * matches code endianness. */ #define RSEQ_SIG_CODE 0xd428bc00 /* BRK #0x45E0. */ - -#ifdef __AARCH64EB__ -#define RSEQ_SIG_DATA 0x00bc28d4 /* BRK #0x45E0. */ -#else #define RSEQ_SIG_DATA RSEQ_SIG_CODE -#endif #define RSEQ_SIG RSEQ_SIG_DATA diff --git a/tools/testing/selftests/seccomp/seccomp_bpf.c b/tools/testing/selftests/seccomp/seccomp_bpf.c index 0622bc2acad40c..79e8eb374cb240 100644 --- a/tools/testing/selftests/seccomp/seccomp_bpf.c +++ b/tools/testing/selftests/seccomp/seccomp_bpf.c @@ -136,6 +136,8 @@ struct seccomp_data { # define __NR_seccomp 354 # elif defined(__x86_64__) # define __NR_seccomp 317 +# elif defined(__alpha__) +# define __NR_seccomp 514 # elif defined(__arm__) # define __NR_seccomp 383 # elif defined(__aarch64__) @@ -1754,6 +1756,30 @@ TEST_F(TRACE_poke, getpid_runs_normally) # define ARCH_REGS struct user_regs_struct # define SYSCALL_NUM(_regs) (_regs).orig_eax # define SYSCALL_RET(_regs) (_regs).eax +#elif defined(__alpha__) +/* NT_PRSTATUS contains r0-r30, pc and unique, not struct pt_regs. */ +struct alpha_regs { + unsigned long regs[33]; +}; + +# define ARCH_REGS struct alpha_regs +/* The kernel keeps the mutable syscall number in r1. */ +# define SYSCALL_NUM(_regs) ((_regs).regs[1]) +# define SYSCALL_RET(_regs) ((_regs).regs[0]) +/* + * Alpha returns positive errno in r0 with the r19/a3 error flag set. + * At a syscall-entry stop, r19 is still the fourth syscall argument. + * Callers must also set the syscall number to -1 to skip the syscall + * before replacing r19 with the error flag. + */ +# define SYSCALL_RET_SET(_regs, _val) \ + do { \ + struct alpha_regs *__regs = &(_regs); \ + long __ret = (_val); \ + \ + __regs->regs[0] = __ret < 0 ? -__ret : __ret; \ + __regs->regs[19] = __ret < 0; \ + } while (0) #elif defined(__arm__) # define ARCH_REGS struct pt_regs # define SYSCALL_NUM(_regs) (_regs).ARM_r7 diff --git a/tools/testing/selftests/timers/clocksource-switch.c b/tools/testing/selftests/timers/clocksource-switch.c index db62a764c29e65..2e86f56d953ebb 100644 --- a/tools/testing/selftests/timers/clocksource-switch.c +++ b/tools/testing/selftests/timers/clocksource-switch.c @@ -40,16 +40,23 @@ int get_clocksources(char list[][30]) { int fd, i; - size_t size; + ssize_t size; char buf[512]; char *head, *tmp; fd = open("/sys/devices/system/clocksource/clocksource0/available_clocksource", O_RDONLY); + if (fd < 0) + return 0; - size = read(fd, buf, 512); + size = read(fd, buf, sizeof(buf) - 1); close(fd); + if (size <= 0) + return 0; + + buf[size] = '\0'; + for (i = 0; i < 10; i++) list[i][0] = '\0'; @@ -74,11 +81,21 @@ int get_clocksources(char list[][30]) int get_cur_clocksource(char *buf, size_t size) { + ssize_t len; int fd; fd = open("/sys/devices/system/clocksource/clocksource0/current_clocksource", O_RDONLY); + if (fd < 0) + return -1; + + len = read(fd, buf, size - 1); + + close(fd); + + if (len <= 0) + return -1; - size = read(fd, buf, size); + buf[len] = '\0'; return 0; } diff --git a/tools/testing/selftests/timers/posix_timers.c b/tools/testing/selftests/timers/posix_timers.c index a92d4b957747ea..52b7289e1450c6 100644 --- a/tools/testing/selftests/timers/posix_timers.c +++ b/tools/testing/selftests/timers/posix_timers.c @@ -113,9 +113,10 @@ static void check_itimer(int which, const char *name) done = 0; - if (which == ITIMER_VIRTUAL) + if (which == ITIMER_VIRTUAL) { + clock_id = CLOCK_THREAD_CPUTIME_ID; signal(SIGVTALRM, sig_handler); - else if (which == ITIMER_PROF) { + } else if (which == ITIMER_PROF) { clock_id = CLOCK_THREAD_CPUTIME_ID; signal(SIGPROF, sig_handler); } @@ -148,7 +149,7 @@ static void check_timer_create(int which) struct itimerspec val = { .it_value.tv_sec = DELAY, }; - int clock_id = CLOCK_REALTIME; + int clock_id = which; timer_t id; done = 0; diff --git a/tools/testing/selftests/timers/raw_skew.c b/tools/testing/selftests/timers/raw_skew.c index 0c87a8fb0d7f9e..dfca3e2d50d2ea 100644 --- a/tools/testing/selftests/timers/raw_skew.c +++ b/tools/testing/selftests/timers/raw_skew.c @@ -91,6 +91,7 @@ int main(int argc, char **argv) { struct timespec mon, raw, start, end; long long delta1, delta2, interval, eppm, ppm; + long tick_nom; struct timex tx1, tx2; setbuf(stdout, NULL); @@ -129,6 +130,17 @@ int main(int argc, char **argv) /* Avg the two actual freq samples adjtimex gave us */ ppm = (long long)(tx1.freq + tx2.freq) * 1000 / 2; ppm = shift_right(ppm, 16); + + /* + * The tick value holds the coarse part of the adjustment, a + * microsecond of the tick length per unit, and time sync daemons + * do put part of their correction there. What it is worth has to + * be counted in as well, or a clock disciplined through it looks + * off by a hundred ppm a unit against what the two clocks show. + */ + tick_nom = USEC_PER_SEC / sysconf(_SC_CLK_TCK); + ppm += (long long)(tx1.tick + tx2.tick - 2 * tick_nom) * + (USEC_PER_SEC / tick_nom) * 1000 / 2; printf(" %lld.%i(act)", ppm/1000, abs((int)(ppm%1000))); if (llabs(eppm - ppm) > 1000) { diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index d478b13cc8d5d9..7565d2cf6a3c93 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -19,7 +19,8 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx lam test_shadow_stack avx apx + corrupt_xstate_header amx lam test_shadow_stack avx apx \ + sigframe_fpu_portability # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall @@ -138,3 +139,5 @@ $(OUTPUT)/avx_64: CFLAGS += -mno-avx -mno-avx512f $(OUTPUT)/amx_64: EXTRA_FILES += xstate.c $(OUTPUT)/avx_64: EXTRA_FILES += xstate.c $(OUTPUT)/apx_64: EXTRA_FILES += xstate.c + +$(OUTPUT)/sigframe_fpu_portability_64: CFLAGS += -mno-avx -mno-avx512f diff --git a/tools/testing/selftests/x86/sigframe_fpu_portability.c b/tools/testing/selftests/x86/sigframe_fpu_portability.c new file mode 100644 index 00000000000000..8377de05203266 --- /dev/null +++ b/tools/testing/selftests/x86/sigframe_fpu_portability.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "helpers.h" +#include "xstate.h" + +#ifndef FP_XSTATE_MAGIC2_SIZE +#define FP_XSTATE_MAGIC2_SIZE sizeof(FP_XSTATE_MAGIC2) +#endif + +/* + * This test verifies the FPU portability and consistency of the signal frame. + * + * - test_valid_shrunk_xstate_size: + * Verifies that the kernel restores state from a frame with xstate_size + * shrunk to only include active features. + * + * - test_invalid_shrunk_xstate_size: + * Verifies that the kernel rejects a frame if xstate_size is too small for + * the features enabled in xfeatures. + */ + +#define SIGFRAME_XSTATE_HDR_OFFSET 512 +#define XSTATE_SSE_ONLY_SIZE (SIGFRAME_XSTATE_HDR_OFFSET + XSAVE_HDR_SIZE) +#define XFEATURE_MASK_FPSSE ((1 << XFEATURE_FP) | (1 << XFEATURE_SSE)) + +static uint32_t ymm_offset; +static uint32_t xstate_size_ymm; +static pid_t self_pid; + +/* + * Load %ymm0 from @v, invoke SYS_kill to deliver @sig, and store the + * restored %ymm0 state back into @v within a single inline assembly + * block so the compiler cannot clobber %xmm0/%ymm0 between steps. + */ +__attribute__((target("avx"))) +static void raise_with_ymm0(int sig, uint64_t *v) +{ + register long rax asm("rax") = SYS_kill; + register long rdi asm("rdi") = self_pid; + register long rsi asm("rsi") = sig; + + asm volatile ("vmovdqu %0, %%ymm0\n\t" + "syscall\n\t" + "vmovdqu %%ymm0, %0" + : "+m" (*(char (*)[32])v), "+r" (rax) + : "r" (rdi), "r" (rsi) + : "rcx", "r11", "ymm0", "memory"); +} + +/* + * Avoid using printf() in signal handlers as it is not + * async-signal-safe. + */ +#define SIGNAL_BUF_LEN 1024 +static char sig_err_buf[SIGNAL_BUF_LEN]; + +static void sig_print(const char *msg) +{ + int left = SIGNAL_BUF_LEN - strlen(sig_err_buf) - 1; + + strncat(sig_err_buf, msg, left); +} + +static void check_avx_support(void) +{ + uint32_t eax, ebx, ecx, edx; + struct xstate_info xstate; + + /* Check CPUID.01H:ECX.OSXSAVE[bit 27] before calling xgetbv to avoid #UD */ + __cpuid(1, eax, ebx, ecx, edx); + if (!(ecx & (1 << 27))) + ksft_exit_skip("OSXSAVE not enabled by OS\n"); + + /* Check XCR0[2] (YMM) is enabled by OS */ + if (!(xgetbv(0) & (1 << XFEATURE_YMM))) + ksft_exit_skip("AVX (YMM) not enabled in XCR0\n"); + + xstate = get_xstate_info(XFEATURE_YMM); + if (!xstate.size) + ksft_exit_skip("AVX not supported by hardware\n"); + + ymm_offset = xstate.xbuf_offset; + xstate_size_ymm = xstate.xbuf_offset + xstate.size; +} + +#define TEST_YMMH_VAL (0x5656565656565656UL) + +static void __handle_shrunk_xstate_size(int sig, siginfo_t *si, void *ucp, bool valid_size) +{ + uint64_t xfeatures, *ymmh_p; + struct xsave_buffer *xbuf; + struct _fpx_sw_bytes *sw; + ucontext_t *uc = ucp; + void *fp; + + fp = uc->uc_mcontext.fpregs; + if (!fp) { + sig_print("fpregs is NULL\n"); + return; + } + + sw = get_fpx_sw_bytes(fp); + if (sw->magic1 != FP_XSTATE_MAGIC1) { + sig_print("magic1 is not valid\n"); + return; + } + + xbuf = (struct xsave_buffer *)fp; + + /* + * Both test cases shrink the frame to contain only AVX (FP + SSE + YMM). + * If valid_size is true, set xstate_size to match the enabled features. + * If valid_size is false, set xstate_size too small (SSE only), which + * the kernel must reject. + */ + if (valid_size) + sw->xstate_size = xstate_size_ymm; + else + sw->xstate_size = XSTATE_SSE_ONLY_SIZE; + + xfeatures = get_xstatebv(xbuf); + xfeatures &= XFEATURE_MASK_FPSSE | (1 << XFEATURE_YMM); + set_xstatebv(xbuf, xfeatures); + set_fpx_sw_bytes_features(fp, xfeatures); + + *(uint32_t *)(fp + sw->xstate_size) = FP_XSTATE_MAGIC2; + + if (valid_size) { + ymmh_p = (uint64_t *)(fp + ymm_offset); + ymmh_p[0] = TEST_YMMH_VAL; + ymmh_p[1] = TEST_YMMH_VAL + 1; + } + + /* clear everything after MAGIC2. */ + if (sw->xstate_size + FP_XSTATE_MAGIC2_SIZE < sw->extended_size) + memset(fp + sw->xstate_size + FP_XSTATE_MAGIC2_SIZE, 0, + sw->extended_size - sw->xstate_size - FP_XSTATE_MAGIC2_SIZE); +} + +static void handle_valid_shrunk_xstate_size(int sig, siginfo_t *si, void *ucp) +{ + __handle_shrunk_xstate_size(sig, si, ucp, true); +} + +static void handle_invalid_shrunk_xstate_size(int sig, siginfo_t *si, void *ucp) +{ + __handle_shrunk_xstate_size(sig, si, ucp, false); +} + +static void test_valid_shrunk_xstate_size(void) +{ + uint64_t v[4]; + + sig_err_buf[0] = 0; + sethandler(SIGUSR1, handle_valid_shrunk_xstate_size, 0); + + v[0] = 0x1111111111111111ULL; + v[1] = 0x2222222222222222ULL; + v[2] = 0x3333333333333333ULL; + v[3] = 0x4444444444444444ULL; + raise_with_ymm0(SIGUSR1, v); + + if (sig_err_buf[0]) + ksft_test_result_fail("%s\n", sig_err_buf); + else if (v[2] == TEST_YMMH_VAL && v[3] == (TEST_YMMH_VAL + 1)) + ksft_test_result_pass("YMM state restored correctly from shrunk frame\n"); + else + ksft_test_result_fail( + "Got upper bits: 0x%lx 0x%lx (expected %lx %lx)\n", + v[2], v[3], TEST_YMMH_VAL, TEST_YMMH_VAL + 1); + + clearhandler(SIGUSR1); +} + +static sigjmp_buf segv_jmpbuf; + +static void handle_segv(int sig, siginfo_t *si, void *ucp) +{ + siglongjmp(segv_jmpbuf, 1); +} + +static void test_invalid_shrunk_xstate_size(void) +{ + uint64_t v[4]; + + sig_err_buf[0] = 0; + sethandler(SIGUSR1, handle_invalid_shrunk_xstate_size, 0); + sethandler(SIGSEGV, handle_segv, 0); + + if (sigsetjmp(segv_jmpbuf, 1) == 0) { + v[0] = 0x1111111111111111ULL; + v[1] = 0x2222222222222222ULL; + v[2] = 0x3333333333333333ULL; + v[3] = 0x4444444444444444ULL; + raise_with_ymm0(SIGUSR1, v); + sig_print("Inconsistent size was NOT rejected\n"); + } + + clearhandler(SIGUSR1); + clearhandler(SIGSEGV); + + if (sig_err_buf[0]) + ksft_test_result_fail("%s\n", sig_err_buf); + else + ksft_test_result_pass("Inconsistent size correctly rejected\n"); +} + +int main(void) +{ + ksft_print_header(); + ksft_set_plan(2); + + self_pid = getpid(); + + check_avx_support(); + + test_valid_shrunk_xstate_size(); + test_invalid_shrunk_xstate_size(); + + ksft_finished(); + return 0; +} diff --git a/tools/testing/selftests/x86/xstate.c b/tools/testing/selftests/x86/xstate.c index 97fe4bd8bc77de..0ab577157cd7d0 100644 --- a/tools/testing/selftests/x86/xstate.c +++ b/tools/testing/selftests/x86/xstate.c @@ -34,18 +34,6 @@ (1 << XFEATURE_XTILEDATA) | \ (1 << XFEATURE_APX)) -static inline uint64_t xgetbv(uint32_t index) -{ - uint32_t eax, edx; - - asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index)); - return eax + ((uint64_t)edx << 32); -} - -static inline uint64_t get_xstatebv(struct xsave_buffer *xbuf) -{ - return *(uint64_t *)(&xbuf->header); -} static struct xstate_info xstate; diff --git a/tools/testing/selftests/x86/xstate.h b/tools/testing/selftests/x86/xstate.h index 6ee816e7625ac7..eedf0cab7ccb7a 100644 --- a/tools/testing/selftests/x86/xstate.h +++ b/tools/testing/selftests/x86/xstate.h @@ -3,6 +3,8 @@ #define __SELFTESTS_X86_XSTATE_H #include +#include +#include #include "kselftest.h" @@ -94,6 +96,14 @@ static inline void xrstor(struct xsave_buffer *xbuf, uint64_t rfbm) : : "D" (xbuf), "a" (rfbm_lo), "d" (rfbm_hi)); } +static inline uint64_t xgetbv(uint32_t index) +{ + uint32_t eax, edx; + + asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index)); + return eax + ((uint64_t)edx << 32); +} + #define CPUID_LEAF_XSTATE 0xd #define CPUID_SUBLEAF_XSTATE_USER 0x0 @@ -160,6 +170,11 @@ static inline void set_xstatebv(struct xsave_buffer *xbuf, uint64_t bv) *(uint64_t *)(&xbuf->header) = bv; } +static inline uint64_t get_xstatebv(struct xsave_buffer *xbuf) +{ + return *(uint64_t *)(&xbuf->header); +} + /* See 'struct _fpx_sw_bytes' at sigcontext.h */ #define SW_BYTES_OFFSET 464 /* N.B. The struct's field name varies so read from the offset. */ @@ -175,6 +190,11 @@ static inline uint64_t get_fpx_sw_bytes_features(void *buffer) return *(uint64_t *)(buffer + SW_BYTES_BV_OFFSET); } +static inline void set_fpx_sw_bytes_features(void *buffer, uint64_t features) +{ + *(uint64_t *)(buffer + SW_BYTES_BV_OFFSET) = features; +} + static inline void set_rand_data(struct xstate_info *xstate, struct xsave_buffer *xbuf) { int *ptr = (int *)&xbuf->bytes[xstate->xbuf_offset]; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 4c58487b764e9d..f89d5d4c0becb3 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -418,8 +418,11 @@ enum { */ enum { READ_IMPLIES_EXEC = 0x0400000, + ADDR_NO_RANDOMIZE = 0x0040000, }; +extern int randomize_va_space; + struct vma_iterator { struct ma_state mas; }; diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h index d6136e19a8af3d..2ca910a7c867b1 100644 --- a/tools/testing/vma/include/stubs.h +++ b/tools/testing/vma/include/stubs.h @@ -27,6 +27,11 @@ struct unmap_desc; #define vma_policy(vma) NULL +static inline u32 get_random_u32(void) +{ + return 0; +} + #define down_write_nest_lock(sem, nest_lock) #define data_race(expr) expr diff --git a/tools/testing/vma/main.c b/tools/testing/vma/main.c index 18338f5d29e00d..381b9fb975d720 100644 --- a/tools/testing/vma/main.c +++ b/tools/testing/vma/main.c @@ -15,6 +15,7 @@ #include "tests/vma.c" int sysctl_max_map_count __read_mostly = DEFAULT_MAX_MAP_COUNT; +int randomize_va_space = 2; /* Helper functions which utilise static kernel functions. */ diff --git a/tools/thermal/tmon/sysfs.c b/tools/thermal/tmon/sysfs.c index cb1108bc924989..5d64aff12a8629 100644 --- a/tools/thermal/tmon/sysfs.c +++ b/tools/thermal/tmon/sysfs.c @@ -225,7 +225,7 @@ static int find_tzone_cdev(struct dirent *nl, char *tz_name, cdev_name, cdev_name_linked, cdev_id); tzi->cdev_binding |= (1 << cdev_id); - /* find the trip point in which the cdev is binded to + /* find the trip point in which the cdev is bound to * in this tzone */ snprintf(cdev_trip_name, sizeof(cdev_trip_name) - 1, diff --git a/tools/thermal/tmon/tmon.c b/tools/thermal/tmon/tmon.c index 7eb3216a27f45f..70a2ec50a46166 100644 --- a/tools/thermal/tmon/tmon.c +++ b/tools/thermal/tmon/tmon.c @@ -171,7 +171,7 @@ static void prepare_logging(void) memset(binding_str, 0, sizeof(binding_str)); for (j = 0; j < 32; j++) - binding_str[j] = (ptdata.tzi[i].cdev_binding & (1 << j)) ? + binding_str[j] = (ptdata.tzi[i].cdev_binding & (1U << j)) ? '1' : '0'; fprintf(tmon_log, "#thermal zone %s%02d cdevs binding: %32s\n", @@ -230,7 +230,7 @@ int main(int argc, char **argv) switch (c) { case 'c': no_control = 0; - strncpy(ctrl_cdev, optarg, CDEV_NAME_SIZE); + snprintf(ctrl_cdev, sizeof(ctrl_cdev), "%s", optarg); break; case 'd': start_daemon_mode(); diff --git a/tools/thermal/tmon/tmon.h b/tools/thermal/tmon/tmon.h index 44d16d778f044c..b7767056f18954 100644 --- a/tools/thermal/tmon/tmon.h +++ b/tools/thermal/tmon/tmon.h @@ -77,7 +77,7 @@ struct tz_info { char type[256]; /* e.g. acpitz */ int instance; int passive; /* active zone has passive node to force passive mode */ - int nr_cdev; /* number of cooling device binded */ + int nr_cdev; /* number of cooling devices bound */ int nr_trip_pts; struct trip_point tp[MAX_NR_TRIP]; unsigned long cdev_binding; /* bitmap for attached cdevs */ diff --git a/tools/thermal/tmon/tui.c b/tools/thermal/tmon/tui.c index 7f5dd2b87f15fb..dbc3ca95c72424 100644 --- a/tools/thermal/tmon/tui.c +++ b/tools/thermal/tmon/tui.c @@ -238,7 +238,7 @@ void show_cooling_device(void) unsigned long trip_binding = ptdata.tzi[i].trip_binding[cdev_inst]; int k = 0; /* per zone trip point id that - * binded to this cdev, one to + * is bound to this cdev, one to * many possible based on the * binding bitmask. */ diff --git a/tools/unittests/test_kdoc_parser.py b/tools/unittests/test_kdoc_parser.py index c4a76ed13dbc7d..4baa78ca02b3f2 100755 --- a/tools/unittests/test_kdoc_parser.py +++ b/tools/unittests/test_kdoc_parser.py @@ -418,6 +418,32 @@ def test_with_default_expected(self): """ self.run_test(self.SOURCE, [self.DEFAULT.copy()], self.EXPORTS) +class TestContextLockStruct(GenerateKdocItem): + def test_context_lock_struct_declaration(self): + source = """ + /** + * struct debugfs_cancellation - cancellation data + * @list: internal, for keeping track + * @cancel: callback to call + * @cancel_data: extra data for the callback to call + */ + context_lock_struct(debugfs_cancellation) { + struct list_head list; + void (*cancel)(struct dentry *, void *); + void *cancel_data; + }; + """ + + kernel_doc = KernelDoc(self.config, "test.c", self.xforms) + patcher = patch('builtins.open', new_callable=mock_open, + read_data=dedent(source)) + with patcher: + _, entries = kernel_doc.parse_kdoc() + + self.assertEqual(len(entries), 1) + self.assertEqual(entries[0].name, "debugfs_cancellation") + self.assertEqual(entries[0].warnings, []) + # # Class and logic to create dynamic tests from YAML # diff --git a/usr/.gitignore b/usr/.gitignore index 8996e7a889024f..13bf7a38355ce5 100644 --- a/usr/.gitignore +++ b/usr/.gitignore @@ -1,4 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -/gen_init_cpio /initramfs_data.cpio /initramfs_inc_data + +# Keep around until cleanup in scripts/remove-stale-files +/gen_init_cpio diff --git a/usr/Kconfig b/usr/Kconfig index 9279a2893ab0ee..f87dd4bac41838 100644 --- a/usr/Kconfig +++ b/usr/Kconfig @@ -13,7 +13,7 @@ config INITRAMFS_SOURCE to be used as an initramfs image. Directories should contain a filesystem layout to be included in the initramfs image. Files should contain entries according to the format described by the - "usr/gen_init_cpio" program in the kernel tree. + "scripts/gen_init_cpio" program in the kernel tree. When multiple directories and files are specified then the initramfs image will be the aggregate of all of them. diff --git a/usr/Makefile b/usr/Makefile index e8f42478a0b765..975c68e5b6b60f 100644 --- a/usr/Makefile +++ b/usr/Makefile @@ -16,8 +16,6 @@ obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o $(obj)/initramfs_data.o: $(obj)/initramfs_inc_data -hostprogs := gen_init_cpio - ramfs-input := $(CONFIG_INITRAMFS_SOURCE) cpio-data := @@ -72,13 +70,13 @@ quiet_cmd_initfs = GEN $@ # 2) There are changes in which files are included (added or deleted) # 3) If gen_init_cpio are newer than initramfs_data.cpio # 4) Arguments to gen_initramfs.sh changes -$(obj)/initramfs_data.cpio: $(src)/gen_initramfs.sh $(obj)/gen_init_cpio $(deps_initramfs) FORCE +$(obj)/initramfs_data.cpio: $(srctree)/scripts/gen_initramfs.sh scripts/gen_init_cpio $(deps_initramfs) FORCE $(call if_changed,initfs) endif $(obj)/initramfs_inc_data: $(cpio-data) FORCE - $(call if_changed,$(compress-y)) + +$(call if_changed,$(compress-y)) targets += initramfs_data.cpio initramfs_inc_data diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 625e62e1a03184..5c084d5393a693 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -506,7 +506,7 @@ static const struct address_space_operations kvm_gmem_aops = { #endif }; -static int kvm_gmem_setattr(struct mnt_idmap *idmap, struct dentry *dentry, +static int kvm_gmem_setattr(const struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *attr) { return -EINVAL;